Version 2.4.1 of the ArcGIS API for Python brought about some exciting new functionality for content management: the arcgis.apps.itemgraph module. This module, in it's most basic usage, allows you to take an item in an organization and examine the deep dependencies of it- what items it needs to exist, what items those items need to exist, and so on. This is very helpful for management of apps and smaller subsets of org items, but what if we really want to push the limits of what we can do with this module?
With a little patience and creativity, we can create an actively-updated dependency graph that encapsulates every item in our organization. Doing so opens up all sorts of doors for us, such as:
- The ability to examine the total reverse dependencies of an item (e.g., take a feature layer- we can see every app or map in our org that requires it to exist, or that will be effected by a change to it
- Quick identification of all broken dependencies in our organization
- Sorting of items in our organization by complexity (most vs. least dependencies)
In this sample, we'll provide code to create and update an organization-wide graph, glean information from our graph, and then finally include a section on how to automate this process via ArcGIS Notebooks.
Part 1: Creating and Updating an Org-Wide Graph
from arcgis.gis import GIS
from arcgis.apps.itemgraph import create_dependency_graph, load_from_file
import datetime
import networkx as nx
gis = GIS("home")Initial Run
This builds the initial dependency graph by indexing over every single item in the organization. This only needs to be run once, and will likely take a very long time, depending on how many items are in the org. Once it's been run and the graph structure and time stamp are saved into files, this notebook that updates the dependency graph can be run. This first cell can also be executed outside of ArcGIS Notebooks, and the .gml and .txt files can be brought in from elsewhere.
Note: Comment out this cell after the first time running the notebook.
# only run this cell the very first time you index over your org and grab all the dependencies
# this will take a long time if the org has a ton of items
import datetime
items = gis.content.search("", max_items = 10000)
timestamp = int(datetime.datetime.now().timestamp()) * 1000
graph = create_dependency_graph(gis, items)
graph.write_to_file("/arcgis/home/graph_data/updating_graph.gml")
with open("/arcgis/home/graph_data/updating_timestamp.txt", "w") as file:
file.write(str(timestamp))Now, we have a snapshot of every item in the organization and how they relate to one another. But, what if we don't want to run this lengthy initial step every time we want to work with this graph? And what if there are items that are getting created, updated, and deleted, so the relationships in this graph might become outdated? This notebook will go over how we can use specific capabilities of the ItemGraph to examine massive graphs without re-indexing, and will include code that will update the graph through automation, meaning that scheduling this notebook to run will ensure that your organization-wide dependency graph stays up to date.
Efficiently Analyzing the Existent Graph
This workflow is predicated on preserving our graph offline in a GML format file, which is a commonly used file type for storing graph structures. Our write_to_file() function generates a .gml file with both the basic graph structure and some additional metadata that informs our load_from_file() function how to grab a given item from a GIS organization.
When loading up an existent graph based on a .gml file, it still may take some time to retrieve every single item associated with each node in the graph, especially for organizations with thousands and thousands of items. If you want to instantly analyze the items and relationships in a .gml file with thousands of records, we can call load_to_file() and elect to not retrieve each item. This still gives us the power to view all item id's and their dependencies in the org, but without the overhead of retrieving each one from a REST endpoint first. Users can simply retrieve the associated items as needed, based on the id's.
We'll show a quick example of this using the existent graph, and then use a graph loaded with the items to ensure we're up-to-date with the current state of organization. We'll only print the first few for the sake of keeping this page short.
fast_graph = load_from_file("/arcgis/home/graph_data/updating_graph.gml", gis, False)
len(fast_graph.all_items())3643
fast_graph.all_items()[:20][ItemNode(id: 72c2671bff1143c4bc577755bc00a7aa), ItemNode(id: 405b512903e74ec0af5ce2083b863705), ItemNode(id: 039e2c7e648b4a5d9d64c57b470d7397), ItemNode(id: 64a096654b924b2998ed07a747524e96), ItemNode(id: e12e8599965c43af9632a5c9633c2994), ItemNode(id: 5b534fa4a26d4fd1868ea7509806b053), ItemNode(id: 96cf6c756ee74c288a8a88788f980266), ItemNode(id: 64480b5bfc474ab1a464fa18098c92e5), ItemNode(id: 14d12cc813b2422db0f4b7b1b701bcc1), ItemNode(id: 5a1a8ac568a942b792c1283b7acff602), ItemNode(id: dc5014dff7014a23af6eff25b492bce6), ItemNode(id: 1fcbf4e77ebc46ef9971d1194f412c3b), ItemNode(id: fc02404cdda24f72842d34b28267d77a), ItemNode(id: 4cc1488a3c134af9abbe0a268d50b499), ItemNode(id: 24ca75398c494dad8b4e7104e2a640d5), ItemNode(id: 4839158f96544bafb841c1739d84f11b), ItemNode(id: 9b9de0ff974e4924b15bc278ad6d6676), ItemNode(id: 87c2394212ad42bcb89de9e244d4072d), ItemNode(id: 13697d5ab62449e78dd9c28037380592), ItemNode(id: a93fd45749ec4773839f02623ebad01d)]
random_node = fast_graph.all_items()[0]
gis.content.get(random_node.id)
Mobile Map Package by esri_nav
As we can see, we can instantly load in this complicated graph and just grab the Item associated with a given node as needed. However, for this next part, we're going to need to retrieve everything in the graph, which could take a while; that's why it's meant for some scheduled automation.
Automation: Updating our Org-Wide Graph
The items in our org might not be the same as the ones we had in it yesterday- that's why we want to update our graph periodically. The ItemGraph class is an extension of NetworkX's DiGraph class, meaning we get all sorts of handy functions and ways to manipulate our graph built in. We can use some attributes of the ItemGraph class and some networkx tricks to maintain an up-to-date record of everything in the org.
We'll start by loading up our existent graph stored in the .gml file with every Item attached. This may take a while to do, depending on how many items must be retrieved. We retrieve the items so that later we can verify which ones still exist and which ones have been deleted.
# grab the graph created yesterday to update it
old_graph = load_from_file("/arcgis/home/graph_data/updating_graph.gml", gis, True)Keeping timestamps allows us to understand when the last time we updated the graph was and maintain an accurate graph. This approach allows for the scheduling of this notebook on any timeframe- whether you run it every day, every week, every month, or something else.
# grab our old and new timestamps
# old is the last time we updated the graph, new is current time
with open("/arcgis/home/graph_data/updating_timestamp.txt", "r") as file:
old_timestamp = int(file.read())
new_timestamp = int(datetime.datetime.now().timestamp()) * 1000
# use these timestamps to see items that have been modified or created since last run
q = f"modified: [{old_timestamp} TO {new_timestamp}]"
modified_items = gis.content.search(query = q, sort_field = "modified", sort_order = "desc")# now that we have our list of updated items, make a graph out of them
mod_graph = create_dependency_graph(gis, modified_items)Now we have a graph containing only new/modified items, and all of their dependencies. We'll have to combine the new information in this graph with our existent graph, and delete the outdated information. Time for some fun graph operations!
# nx.compose() combines all nodes and relationships of two graphs
new_graph = nx.compose(old_graph, mod_graph)
# find the list of items in both graphs
# these are ones that already existed but were modified, meaning dependencies might have changed
intersects = set(old_graph.all_items('id')) & set(mod_graph.all_items('id'))
# go through these items and delete outdated forward dependencies
for itemid in list(intersects):
old_contains = set(old_graph.get_node(itemid).contains("id"))
mod_contains = set(mod_graph.get_node(itemid).contains("id"))
# isolate deps that only existed in old graph
outdated = old_contains - mod_contains
# remove these relationships from the new graph
for dep in list(outdated):
new_graph.delete_relationship(itemid, dep)Our new_graph now has an accurate reflection of all item relationships in our organization- any item that got changed could have had it's forward dependencies altered, so we made sure that those got updated to reflect any changes. We didn't have to worry about changing any reverse dependencies, since every reverse dependency is still just a forward dependency for another item, so the appropriate relationship was addressed during that process.
We have one more little housekeeping thing to do with this new graph now- due to the way that nx.compose() works, the individual nodes within the new graph still contain references to the old graph. We just have to make sure those nodes now use the new graph for any computation.
# make sure our nodes are all referencing the new updated graph
for node in new_graph.all_items():
node.graph = new_graphAlright, awesome, now we have all newly created or recently modified items accounted for- but we still have to account for items that might have been deleted. How do we do this?
Remember when we ran the load_from_file() function earlier that retrieved all of the items while rebuilding the graph? Well, any node of an item that's been deleted won't have an attached Item object anymore, since it wouldn't have been retrieved. Therefore, if we have a node with no associated item, we can assume the item has been deleted, and that if the node is not contained by another node, then the item is not still meant to exist. So, we can delete nodes from our graph that fit that criteria.
One added layer of complexity: what if the node of a deleted item is contained by another node of a deleted item that just hasn't been processed yet? We can avoid this scenario by sorting our nodes in ascending order of how many items they're required by, ensuring we're prioritizing processing ones furthest up the dependency chains first (e.g., deleted item A contains B which contains C- if we analyze B or C first, we'll wrongfully keep them, but if we start with A and go down the chain, all will get deleted properly).
nodes = new_graph.all_items()
# define our sort function
def count_reqs(node):
return len(node.required_by("id"))
# this orders our nodes properly
sorted_nodes = sorted(nodes, key=count_reqs)
# remove ones that don't have items and aren't dependencies of others
for node in sorted_nodes:
if not node.item and not node.contained_by():
new_graph.remove_node(node.id)And there we have it. Now we have a properly updated graph, one that reflects today's state of every item and their dependencies. Grabbing a node for a given item from new_graph should give you a holistic look of all items it relates to in an org, no matter if it's an immediate or deep relationship in either direction. To summarize, we:
- Loaded in the graph from the last time we ran the notebook
- Determined what items had been modified/created since the last time we ran the notebook
- Created a graph from those updated items, capturing their current dependencies
- Merged the old graph with the modified graph to create a new graph with all nodes/relationships from both graphs
- Got rid of relationships that no longer existed from the new graph
- Got rid of any items that had been deleted and weren't dependencies of an item from the new graph
Now, we'll write out the state of this graph and the timestamp at which we started everything, so we can repeat this process in the future and continue to update our graph as the items in our ArcGIS organization evolve.
new_graph.write_to_file("/arcgis/home/graph_data/updating_graph.gml")
with open("/arcgis/home/graph_data/updating_timestamp.txt", "w") as file:
file.write(str(timestamp))Part 2: Using the Graph
Now that we have our organization-wide graph and know that it's up to date, we can use it for some useful administration and analysis. We'll take a look at the three things mentioned in the intro: examining comprehensive reverse dependencies of items, identifying broken dependencies that need fixing, and sorting our items based on their complexity.
Comprehensive Reverse Dependencies
This is a very powerful perk of having every relationship in an organization included in a graph: the ability to see every single item that relies on a specific item. Say you have a feature layer that is used in numerous maps and applications throughout your org: you can instantly acquire a list of every one that consumes it, either directly or indirectly. Let's take a look.
# grab a node corresponding to a feature layer
flayer_id = "7dc6cea0b1764a1f9af2e679f642f0f5"
flayer_node = new_graph.get_node(flayer_id)
flayer_node.item
Calling .contained_by() will show all of the items that contain our vector tile layer.
flayer_node.contained_by("item")[<Item title:"National Day1 - With Created Layer" type:Web Map owner:arcgis_python>, <Item title:"National Day1 - With Created Layer" type:Web Map owner:arcgis_python>, <Item title:"AZ and NM Cities" type:Web Map owner:nparavicini>, <Item title:"Topographic" type:Web Map owner:nparavicini>, <Item title:"Living Atlas Map" type:Web Map owner:nparavicini>, <Item title:"Capital Map" type:Web Map owner:nparavicini>, <Item title:"New York Enriched Top Areas of Interest" type:Web Map owner:arcgispyapibot>, <Item title:"New York Enriched Top Areas of Interest" type:Web Map owner:arcgispyapibot>]
Calling .required_by() will show us all of the items that depend on our layer in some way.
flayer_node.required_by("item")[<Item title:"All About Arizona and New Mexico" type:StoryMap owner:nparavicini>, <Item title:"National Day1 - With Created Layer" type:Web Map owner:arcgis_python>, <Item title:"Living Atlas Map" type:Web Map owner:nparavicini>, <Item title:"Ruff Ruff" type:StoryMap owner:nparavicini>, <Item title:"Capital Map" type:Web Map owner:nparavicini>, <Item title:"New York Enriched Top Areas of Interest" type:Web Map owner:arcgispyapibot>, <Item title:"Topographic" type:Web Map owner:nparavicini>, <Item title:"AZ and NM Cities" type:Web Map owner:nparavicini>, <Item title:"National Day1 - With Created Layer" type:Web Map owner:arcgis_python>, <Item title:"Copy this story" type:StoryMap owner:arcgis_python>, <Item title:"New York Enriched Top Areas of Interest" type:Web Map owner:arcgispyapibot>, <Item title:"San Diego Parrot Sightings" type:Form owner:nparavicini>, <Item title:"Testing Copy Content" type:StoryMap owner:arcgispyapibot>]
There we have it- we instantly determined which items would be affected by deleting or changing our layer.
Finding Broken Dependencies
In this context, we're calling an item that is contained by another item, but isn't valid or accessible, a "broken" dependency- these sorts of items can lead to rendering or logic issues in applications such as Web Maps, StoryMaps, Experiences, Surveys, and so on. There can be a number of reasons for why these improper dependencies may exist- an item being used by an application mistakenly got deleted, there was some sort of corruption with data, the json structure of an application might have been copied from one in a different org, and so on.
Luckily, the way the ItemGraph is constructed allows us to instantly find these cases where something is potentially awry. During construction of the graph, the ID of each ItemNode was passed into the GIS org to see if it yielded an item. Therefore, any node that doesn't have an .item property could potentially be a problem.
no_item_nodes = [node for node in new_graph.all_items() if not node.item]
no_item_nodes[ItemNode(id: 370b95fdf676475ca8fd184ad0931610), ItemNode(id: 1dc6042bb3ba409c9d1eb1d7c996d6ed), ItemNode(id: bf08d46ab087435aa4afbc62092223cb), ItemNode(id: 123hehethiswontwork), ItemNode(id: 2fe20a161c514e16b805a21a7e2d636b), ItemNode(id: 15efb8c0d9e3447782789592f432fc1a), ItemNode(id: bda2f1dc3c824c3e97f4a682554177cf), ItemNode(id: bbf4d26d7c3340dab009c7164cc3e1e5), ItemNode(id: d60affb34dd948a1b973da076830ad00), ItemNode(id: 1182ef6f62a543fdba4a55c85c014431), ItemNode(id: f32924d7b16f41d78839dba3b943b08c), ItemNode(id: 9f91d4c34f194da793c8573b24a751a9), ItemNode(id: 91fa101048bb4855970f7d004a4542ce), ItemNode(id: 6e02b538bea841ed858ef9f52709b655), ItemNode(id: 878802f78aad4ab5a83c1397c72048c5), ItemNode(id: f1801a1b3d674de7b9de246a126eb17b), ItemNode(id: 2), ItemNode(id: 44c3348139a84423b4a46074a104efdf), ItemNode(id: 74ab88e7e22d4dbd93b43a00b2d0155b), ItemNode(id: 306cf223a2fd46d3bf6ce39225681977), ItemNode(id: 1ee3200825ce448c91a1ec9ae4f70c28), ItemNode(id: 72224efc9e044001934bb3be87120beb), ItemNode(id: 90ff936dc4704a0798f61ebbb1dec357), ItemNode(id: 0df19d2d940b47658857cc2fc9623840), ItemNode(id: cdf23b869e7649469759cf756c1d43f0), ItemNode(id: a5f11273b85d40adbd7cf9a162f81ef2), ItemNode(id: 08a9b819b9e04de9a68418f2b225ec6a), ItemNode(id: f265c79c47a3413b8b8f711909f03fe2), ItemNode(id: 819ee7ddbf5848889effd0fa6b039a51), ItemNode(id: 8548e23d8b1a478eaf2246e6fc3b31b9), ItemNode(id: f97edbe48af7427f8604f20320e2de52), ItemNode(id: b66792b3bb004fa8997fcfcef991e984), ItemNode(id: 9204269c8e714500a565a3ddf055825d), ItemNode(id: 28dae0b8f1c44814b281b71752453c61), ItemNode(id: d15c117e60dd4fe299ec03a379c9caf8), ItemNode(id: 3b6d38a456164adea4c061a2a51d89b1), ItemNode(id: a12fcbe07c8d41f1b6981f33a10d1d0c), ItemNode(id: ae4bea7c35824bf1ae571783142dc97d), ItemNode(id: 8e969d9adb7f485c88a8350d317e8c9b), ItemNode(id: 493db5d6832b44a581547a8df8c3907c), ItemNode(id: ebe640ee5d344bea9fca7bfa4d80bb3a), ItemNode(id: 6595a5cf08d64212a85ce9dc47700575), ItemNode(id: 761e9fcf5efa4b368716fae9a1ffda0f), ItemNode(id: 39f0fc3c5b424d679c2ef9ae67e71624), ItemNode(id: 59835c8d71d94475ae04f540a8167343), ItemNode(id: 5b2e49a83dd9427abf398aeeecc00f75), ItemNode(id: dcf4a23abb614d7db9cda2dce90cc528), ItemNode(id: 1d250ce2c5c0463abe0e1887597cc926), ItemNode(id: e57b96c732cc4c34830756d53c8f3619), ItemNode(id: 79f3399102ce4ab79bed0407b03005fc), ItemNode(id: 872d78b038d342b98b6a3547ed8a7ac9), ItemNode(id: 3a2d7671322e4dcbb386e4b961e28c27), ItemNode(id: 8900b18d92ac43cfae142a5a5ad91e4a), ItemNode(id: ba6575ad916041a89887309492473b42), ItemNode(id: 5bbbe8978d9f43a0a7a947b5af3cdc1d), ItemNode(id: 73196ca49b6b40c0a03206663cb10554), ItemNode(id: 9566f596ba404a05aa8eae9221bce82a), ItemNode(id: c2e2b6eaea3740059546951c16af5151), ItemNode(id: 4cee55fa5973406596359e7630ab167e), ItemNode(id: 086eb0c799794d4b99a95a39cd43dca2), ItemNode(id: 6313031ddbe14d7ca0358c4ba8011b27), ItemNode(id: bc68a4d142fb4801a2401a0a5a551ba1), ItemNode(id: 07b7fe74b02c40afa1551d41f19f6d99), ItemNode(id: 632d3de849224fcb990f6d9feae36331), ItemNode(id: 26e87527293b43978b1fb4a24133ead1), ItemNode(id: efff5de20fd546de94e613fbed730372), ItemNode(id: 401d890c10d94e88948a0bd5f82e7559), ItemNode(id: 8cf058d7e58d4f66831e8bc7ae7addbd), ItemNode(id: fed737361fcf4815b2b62651e1cd2a57), ItemNode(id: 46472e81856c4420aeb091e6f0fd24e8), ItemNode(id: d1414cb87c2d41099cfcfe55aa8344e2), ItemNode(id: c2f21b0e86de49fa9b6437cd72779488), ItemNode(id: 77efb9bd7831430ab3788741d327e5df), ItemNode(id: 633c43f3c0e148dea129d6ca49e7fd22), ItemNode(id: 3dce15e4178a4a9c92cacb66f2f7f1ea), ItemNode(id: 2a6e5de8aa4a40899e44017121c5cd89), ItemNode(id: bccbd571d0d54e45b6a69ca0249e7132), ItemNode(id: c8cbf786be774df7b064f149c5898fef), ItemNode(id: 71276e9e9638459890a3072df64e31b6), ItemNode(id: a06302053b3f4e35961584d5818bf1f2), ItemNode(id: 5feee1e7f41a4c47b3f137457c130182), ItemNode(id: 6e403956c5d7419b9c0628e65bed0769), ItemNode(id: 196b5b2798204de38b6dd125823bd714), ItemNode(id: 88ba2cd25d1c423c808fa2e1941a540b), ItemNode(id: c1e6eb63502c4012bb631097b69aefad), ItemNode(id: 4c56c5e083544decbd8d61f3a0bc5a67), ItemNode(id: 905567641b10499ba6a77a0a6e4e5f13), ItemNode(id: 861096f2cb274134bcdca7e875899d69), ItemNode(id: 3afb37e1c7784a05b7d0b4c4c98088c1), ItemNode(id: ae5179630d714200893cda3a6e50e460), ItemNode(id: 2243cb05a01f43cc8d7439447ffe3bbf), ItemNode(id: 27d2f0ad1c7545e48848b43bbd51581a), ItemNode(id: 9dc85299a77540b0bac1507ce67e9dfa), ItemNode(id: 16ffb93988c94d699b672d837c1e03e6), ItemNode(id: 3668c924551b4ff2a508cd2cda897abe), ItemNode(id: c2e075d098a6493393dc3f55876d48f4), ItemNode(id: 53071c1408d240c1997a3779c48467b4), ItemNode(id: 36a01f823259430197c896ff415ee493), ItemNode(id: 21e0d7322928434d8889eba70e4c645a), ItemNode(id: 3e5ba2fec29f4f25afd5a5dbfaadf136), ItemNode(id: d79cd0205b03479b8019648e1dd057f4), ItemNode(id: 8461fdc33b554ffbad1ba54afd1ff499), ItemNode(id: 940e3b1543bf4289822790a15e66c5a4), ItemNode(id: 15e70e3da80b42fd9b60eda8cd6e7cd4), ItemNode(id: 4438ff8fb9174783934412c1c51f4865), ItemNode(id: db089afa931a4a9fb1cb2afe33a6c2f6), ItemNode(id: ba0ff62d9aaf4f628eb1a869c0bb5ee0), ItemNode(id: d50ecf25a35a4cd9baaa1ba7035f77da), ItemNode(id: 76e505d722f64fa6b0debd0f196c3ae1), ItemNode(id: dfcdd3b0495d4f4cb3c3a4a401616d6e), ItemNode(id: 03bcd4dfbeae4a91ae270a934fa27e1b), ItemNode(id: 1c5e01156f5a4f0fae4dc61297785c1d), ItemNode(id: 16d1d5083719466487007bf90313184f), ItemNode(id: 2f87efbdf4ec4942ad610fe4b6bd8c36), ItemNode(id: 97162f48110d4458820c3b66de375d67), ItemNode(id: e98bc26706ba46c782c03c81c739a065), ItemNode(id: 8d721aaa69aa4715911693fa4863b7cf), ItemNode(id: 7a238f29b99f4ff1957e3a469177c56a), ItemNode(id: ff9cafd475ff4a0abf5d8f3a34fc7646), ItemNode(id: ae00236efed94d0b8739ef8a244cc312), ItemNode(id: 12b7fd7eaefc4f40b2fe003604153b3c), ItemNode(id: 1f2248e4464b4065a864cc6aab7aa780), ItemNode(id: c13634e58a024cae9de6a15a4b3eb101), ItemNode(id: bd6943458948460a8eb71d97baaa2ada), ItemNode(id: 6fafb84fc46d4b9c8b25fb57dd9911a2), ItemNode(id: c209e14b44b7420cb40ed650cfefa4f5), ItemNode(id: 048de0545a834d8b8c31a6e0e10270d7), ItemNode(id: 40bf682bf17c4c32a5acade2d54eafb9), ItemNode(id: e608d5be1b4241cdbb1fbf61418513d6), ItemNode(id: 908ceeaa2e5e4ef19e823715a35ab978), ItemNode(id: 7d907c5ac31b4da098d47922bb31f2aa), ItemNode(id: 5dd87bebb9864dac9c2f42604bb79b14), ItemNode(id: 78186ce321f1433091aeb2593abbc565), ItemNode(id: cb7a89a516e84af7b3d7e551a38afed5), ItemNode(id: 6b63dafabf4e486183c21bede31b149e), ItemNode(id: a962b03a45d9415ea97b96eaab7f20d6), ItemNode(id: ac96d6152ba34373ac1c7288b2fcefe4), ItemNode(id: d0d5d3caafed43cd9ce18b9a2609fc5f), ItemNode(id: d9b8675be48d49078db4c55208a0d19e), ItemNode(id: 2defd52c9e4f4759ba085f339d284cd4), ItemNode(id: 961434231c034930801f5b3237715587), ItemNode(id: af83304d94cd4e0da83318f76cde60c4), ItemNode(id: 85179dd8a0fe4ea597586fed851bdeff), ItemNode(id: d5e20e6f7a9542d5ace62983699d3212), ItemNode(id: e3e1712a046d4feeb454e5538a6049d7), ItemNode(id: 045ba64dd5bc4b1a94350f897de956cf), ItemNode(id: 4fb8f3f5422d4a7696b0f8136a4b48aa), ItemNode(id: 75113e1eb7e045229c2cb2698e43114c), ItemNode(id: 228834a1eff64c629c80d0b638f9e43c), ItemNode(id: 42b653e394104606987cbf4b6d1587f9), ItemNode(id: 2c82e7cf420b4026b875ad45750c7a24), ItemNode(id: f1894e0262e44dd6adadef875577d8aa), ItemNode(id: 4f93f1fbebdd4f66a55b6a02f332fa20), ItemNode(id: caa4a0e4ddcb47c9845eb92f024fcd7f), ItemNode(id: 1a9c7b34467d41728f9472a0b6628841), ItemNode(id: 7f4342f006a44c8780d8df79d5da5652), ItemNode(id: a90891b098034e0d928fc4ee30dec1a3), ItemNode(id: caa7ab7f82ab41dd9af1215c11f55a8f), ItemNode(id: bb46d017f7af49728d5af8d8eec43c4e), ItemNode(id: a5df7d4a77c24fc39a1115122370cd8f), ItemNode(id: 879921e888a14e96b5f5b339af907acd), ItemNode(id: 8dd1b1b7731c49839489e0d30146f513), ItemNode(id: eb0499aeb1d5461f877e9b40534e225e), ItemNode(id: 4138c0fbe6b446fbad5447ecc34c8ab8), ItemNode(id: e2ee852f11c14d99814dd4ed42a0a16c), ItemNode(id: e27013ac9f664054a044edd3567c763a), ItemNode(id: ee527fed605a4088b10c49ab9c9ed096), ItemNode(id: a7cdce7ebb404ae6b700b491f21b3243), ItemNode(id: e0dfd02f8f61479c8e1a276c79cafcd5), ItemNode(id: 3d0d28a295e54d25966e7945e2bf5d8e), ItemNode(id: d8e9dfe815b44c8ab3e94a6dc1fcf201), ItemNode(id: 3ce93787d12645f386d42d57ca57c1b4), ItemNode(id: 9c03b76829464ee68d4e6af241cebd5e), ItemNode(id: 5707b86b5f3f4b7c9e67ac2fd317254a), ItemNode(id: 79719a8a27d3440d9d622041ff7b8506), ItemNode(id: 1adc07ec24774989873cafc6c68c1536), ItemNode(id: 1926c33e9bf34ef99b8950f1408368a3), ItemNode(id: 892a696bf2aa4a899f5177017d998156), ItemNode(id: 57654725661049eba50bca930ae786e8), ItemNode(id: c85179865e0c45089e0f9bb29b7ca83e), ItemNode(id: b9cb2d66aaa04093950683767c8eaff0), ItemNode(id: c9b04902a93c4aeda9b92e45b303fe01), ItemNode(id: 1c8354e6f69e4df1969befae339412c7), ItemNode(id: ed2246c081ee49369024cfb312e34899), ItemNode(id: 6d6bae1a02484353add60ddefaca185e), ItemNode(id: a143d4fe84e14712bfd28e71b26ac868), ItemNode(id: d822827df8f345d280ebf2c2be8a48a1), ItemNode(id: 3f4789f0a58b4cdcab5b7d1a08f5f9d9), ItemNode(id: f091f6195a194f38a47f8b329bbd9255), ItemNode(id: 7f7c4245f03445e782e9aa4a4d06221d), ItemNode(id: 07a05ca8a10843aba797076d40ca8f36), ItemNode(id: https://oceans1.arcgis.com/arcgis/rest/services/USA_Drilling_Platforms/FeatureServer/0), ItemNode(id: e26f2752e86d4a8dad52ea75501480f0), ItemNode(id: aecf7fd35dca43fe80fc67b60fb73516), ItemNode(id: ab924202cc404af58b41efcf99b0aafa), ItemNode(id: 1c735ebaed8d48b5bfd6fcfcd35eb3d1), ItemNode(id: a5846d8e82834f4398fb54e0ef0db0c5), ItemNode(id: bbf872949f2f43a6921e716f7fbe1d72), ItemNode(id: 1c80207bf5384974994ad4a13abf4556)]
As we can see, we now have a list of nodes where the ID's don't lead to a portal item. There is one anomaly here: what appears to be a service that just doesn't have an associated portal item, but still may be valid. We can add some separate logic to ensure we test any URL for valid data.
from arcgis.layers import Service
updated_no_item = []
for node in no_item_nodes:
if "https://" in node.id:
try:
serv = Service(node.id)
if not serv:
raise
except:
updated_no_item.append(node)
else:
updated_no_item.append(node)Given this list of items with ID's that lead nowhere, we can grab the node for any one of them and do what we did above- see which org items contain or require them. From there, we can figure out how to fix it- one option is using the remap_data() function, documented here.
broken_node = updated_no_item[0]
broken_node.contained_by()[ItemNode(id: e6430fd93abe46ccaf2a8e31b5a35c80, item: AZ Vehicle Incidents)]
broken_node.required_by()[ItemNode(id: e6430fd93abe46ccaf2a8e31b5a35c80, item: AZ Vehicle Incidents), ItemNode(id: d3b24b91f2c3432fbb5b7d2862f7525d, item: TestInstantApp), ItemNode(id: 7259119a69044ae4aebd3e4cd338321f, item: AZ Vehicle App)]
Sorting Items by Degree, Type, and More
Finally, let's put ourselves in the shoes of an organization admin that wants to learn about the items in their org: which items have the most dependencies, and which ones are needed by the most items? Which ones have the most broken dependencies? Let's take a look at how we can capitalize on some graph logic and networkx tricks to gain insights into our organization's content.
We'll start basic: just identify which nodes have the highest degree, or number of direct relationships ("contains"/"contained by"). Since a degree can refer to either direction relationship, we specify the out degree or in degree, which in an ItemGraph will show the amount of each node's contains() and contained_by(), respectively.
def sort_degrees(direction = "out", descending = True, result_count = 50):
node_degrees = new_graph.out_degree()
if direction == "in":
node_degrees = new_graph.in_degree()
nodes_sorted = sorted(node_degrees, key=lambda item: item[1], reverse=descending)
if result_count and result_count > len(nodes_sorted):
result_count = len(nodes_sorted)
for node, degree in nodes_sorted[:result_count]:
graph_node = new_graph.get_node(node)
if graph_node.item:
print(f"Title: {graph_node.item.title}, Type: {graph_node.item.type}, Degree: {degree}")
else:
print(f"ID: {node}, Degree: {degree}")# find which items contain the most other items
sort_degrees(direction = "out", descending = True, result_count = 50)Title: New York Enriched Top Areas of Interest, Type: Web Map, Degree: 158 Title: New York Enriched Top Areas of Interest, Type: Web Map, Degree: 158 Title: DevTechSummit2025-copy-1745540837450, Type: Hub Site Application, Degree: 13 Title: My Site in Enterprise - new, Type: Hub Site Application, Degree: 12 Title: San Diego Parrot Sightings, Type: Form, Degree: 6 Title: Discover Green Spaces and Wildlife in Berlin, Type: Web Mapping Application, Degree: 6 Title: AZ Vehicle Incidents, Type: Web Map, Degree: 4 Title: Living Atlas Map, Type: Web Map, Degree: 4 Title: SimpleWebMap, Type: Web Map, Degree: 4 Title: Arkansas Hospitals, Type: Web Map, Degree: 4 Title: TestInstantApp, Type: Web Mapping Application, Degree: 3 Title: map test, Type: Web Map, Degree: 3 Title: Cycling Explorer, Type: Web Map, Degree: 3 Title: AZ and NM Cities, Type: Web Map, Degree: 3 Title: Jy Survey 123 Connect, Type: Form, Degree: 3 Title: Jy Survey 123 Connect, Type: Form, Degree: 3 Title: map test, Type: Web Map, Degree: 3 Title: Group Layer Map Test, Type: Web Map, Degree: 3 Title: Map Series 884411, Type: Web Mapping Application, Degree: 3 Title: Map Series 213975, Type: Web Mapping Application, Degree: 3 Title: Map Series 959005, Type: Web Mapping Application, Degree: 3 Title: Map Series 537579, Type: Web Mapping Application, Degree: 3 Title: Map Series 845552, Type: Web Mapping Application, Degree: 3 Title: Map Series 836403, Type: Web Mapping Application, Degree: 3 Title: Map Series 639470, Type: Web Mapping Application, Degree: 3 Title: Map Series 816182, Type: Web Mapping Application, Degree: 3 Title: Map Series 398285, Type: Web Mapping Application, Degree: 3 Title: Map Series 985608, Type: Web Mapping Application, Degree: 3 Title: Map Series 287154, Type: Web Mapping Application, Degree: 3 Title: Map Series 396347, Type: Web Mapping Application, Degree: 3 Title: Map Series 684806, Type: Web Mapping Application, Degree: 3 Title: Map Series 709821, Type: Web Mapping Application, Degree: 3 Title: Map Series 587161, Type: Web Mapping Application, Degree: 3 Title: Map Series 255073, Type: Web Mapping Application, Degree: 3 Title: Map Series 553493, Type: Web Mapping Application, Degree: 3 Title: Map Series 329174, Type: Web Mapping Application, Degree: 3 Title: Map Series 37166, Type: Web Mapping Application, Degree: 3 Title: Map Series 261798, Type: Web Mapping Application, Degree: 3 Title: Map Series 395417, Type: Web Mapping Application, Degree: 3 Title: Map Series 326103, Type: Web Mapping Application, Degree: 3 Title: Map Series 116930, Type: Web Mapping Application, Degree: 3 Title: Map Series 222445, Type: Web Mapping Application, Degree: 3 Title: Map Series 432118, Type: Web Mapping Application, Degree: 3 Title: USA WebMap, Type: Web Map, Degree: 3 Title: AZ Vehicle App, Type: Web Mapping Application, Degree: 3 Title: Map Series 692346, Type: Web Mapping Application, Degree: 3 Title: Map Series 772573, Type: Web Mapping Application, Degree: 3 Title: Map Series 254278, Type: Web Mapping Application, Degree: 3 Title: Map Series 19528, Type: Web Mapping Application, Degree: 3 Title: Map Series 238550, Type: Web Mapping Application, Degree: 3
# find which items are contained by the most other items
sort_degrees(direction = "in", descending = True, result_count = 50)ID: bf08d46ab087435aa4afbc62092223cb, Degree: 153 ID: 123hehethiswontwork, Degree: 153 ID: 7f7c4245f03445e782e9aa4a4d06221d, Degree: 33 ID: 07a05ca8a10843aba797076d40ca8f36, Degree: 33 Title: World Topographic Map, Type: Vector Tile Service, Degree: 8 Title: Web Map 400800, Type: Web Map, Degree: 7 Title: Outline Map, Type: Web Map, Degree: 7 Title: Newspaper (WGS84), Type: Vector Tile Service, Degree: 5 Title: OpenStreetMap Streets with Relief, Type: Vector Tile Service, Degree: 5 ID: 2, Degree: 3 Title: Esri_Regional_Offices_US, Type: File Geodatabase, Degree: 3 Title: Jy Survey 123 Connect, Type: Feature Service, Degree: 3 Title: Jy Survey 123 Connect, Type: Feature Service, Degree: 3 Title: AZ Vehicle Incidents, Type: Web Map, Degree: 2 Title: Basic, Type: Web Mapping Application, Degree: 2 Title: AGO World Geocoder (arcgis_python), Type: Geocoding Service, Degree: 2 ID: 2fe20a161c514e16b805a21a7e2d636b, Degree: 2 ID: 15efb8c0d9e3447782789592f432fc1a, Degree: 2 ID: bda2f1dc3c824c3e97f4a682554177cf, Degree: 2 ID: d60affb34dd948a1b973da076830ad00, Degree: 2 ID: f32924d7b16f41d78839dba3b943b08c, Degree: 2 ID: 9f91d4c34f194da793c8573b24a751a9, Degree: 2 ID: 91fa101048bb4855970f7d004a4542ce, Degree: 2 ID: 6e02b538bea841ed858ef9f52709b655, Degree: 2 Title: National Weather Service Day 1, Type: WMS, Degree: 2 Title: Enhanced Contrast Dark Map (WGS84), Type: Web Map, Degree: 2 ID: 44c3348139a84423b4a46074a104efdf, Degree: 2 Title: Test Case 12305 Survey, Type: Feature Service, Degree: 2 Title: Test Case 232 A, Type: Feature Service, Degree: 2 Title: USA_Major_Cities_Data, Type: Feature Service, Degree: 2 Title: San Diego Parrot Sightings, Type: Feature Service, Degree: 2 Title: DevSummit Theme, Type: StoryMap Theme, Degree: 2 Title: ca_public_schools, Type: Feature Service, Degree: 2 ID: cdf23b869e7649469759cf756c1d43f0, Degree: 2 ID: a5f11273b85d40adbd7cf9a162f81ef2, Degree: 2 ID: 08a9b819b9e04de9a68418f2b225ec6a, Degree: 2 ID: f265c79c47a3413b8b8f711909f03fe2, Degree: 2 ID: 819ee7ddbf5848889effd0fa6b039a51, Degree: 2 ID: 8548e23d8b1a478eaf2246e6fc3b31b9, Degree: 2 ID: f97edbe48af7427f8604f20320e2de52, Degree: 2 ID: b66792b3bb004fa8997fcfcef991e984, Degree: 2 ID: 9204269c8e714500a565a3ddf055825d, Degree: 2 ID: 28dae0b8f1c44814b281b71752453c61, Degree: 2 ID: d15c117e60dd4fe299ec03a379c9caf8, Degree: 2 ID: 3b6d38a456164adea4c061a2a51d89b1, Degree: 2 ID: a12fcbe07c8d41f1b6981f33a10d1d0c, Degree: 2 ID: ae4bea7c35824bf1ae571783142dc97d, Degree: 2 ID: 8e969d9adb7f485c88a8350d317e8c9b, Degree: 2 ID: 493db5d6832b44a581547a8df8c3907c, Degree: 2 ID: ebe640ee5d344bea9fca7bfa4d80bb3a, Degree: 2
We can similarly define a function that will do that for the "deep" degrees- finding which items have the most recursive dependencies or items dependent upon them.
def sort_deep_degrees(direction = "out", descending = True, result_count = 50):
if direction == "in":
nodes_weighted = [(node, len(node.required_by())) for node in new_graph.all_items()]
else:
nodes_weighted = [(node, len(node.requires())) for node in new_graph.all_items()]
nodes_sorted = sorted(nodes_weighted, key=lambda item: item[1], reverse = descending)
if result_count and result_count > len(nodes_sorted):
result_count = len(nodes_sorted)
for node, degree in nodes_sorted[:result_count]:
if node.item:
print(f"Title: {node.item.title}, Type: {node.item.type}, Degree: {degree}")
else:
print(f"ID: {node.id}, Degree: {degree}")sort_deep_degrees()Title: Testing Copy Content, Type: StoryMap, Degree: 159 Title: New York Enriched Top Areas of Interest, Type: Web Map, Degree: 158 Title: New York Enriched Top Areas of Interest, Type: Web Map, Degree: 158 Title: DS-Page1-copy-1745540879568, Type: Hub Page, Degree: 14 Title: DevTechSummit2025-copy-1745540837450, Type: Hub Site Application, Degree: 13 Title: My Site in Enterprise - new, Type: Hub Site Application, Degree: 12 Title: TestInstantApp, Type: Web Mapping Application, Degree: 8 Title: San Diego Parrot Sightings, Type: Form, Degree: 8 Title: All About Arizona and New Mexico, Type: StoryMap, Degree: 8 Title: AZ Vehicle App, Type: Web Mapping Application, Degree: 8 Title: Discover Green Spaces and Wildlife in Berlin, Type: Web Mapping Application, Degree: 7 Title: Formerly Local, Now Global Experience, Type: Web Experience, Degree: 6 Title: AZ Vehicle Incidents, Type: Web Map, Degree: 5 Title: AZ and NM Cities, Type: Web Map, Degree: 5 Title: Ruff Ruff, Type: StoryMap, Degree: 5 Title: Group Layer Map Test, Type: Web Map, Degree: 5 Title: LimerickStoryMap, Type: StoryMap, Degree: 5 Title: test_cat_selector, Type: Dashboard, Degree: 5 Title: Copy this story, Type: StoryMap, Degree: 4 Title: map test, Type: Web Map, Degree: 4 Title: Cycling Explorer, Type: Web Map, Degree: 4 Title: Offline Cloning Draft Test, Type: Web Experience, Degree: 4 Title: Jy Survey 123 Connect, Type: Form, Degree: 4 Title: Living Atlas Map, Type: Web Map, Degree: 4 Title: Jy Survey 123 Connect, Type: Form, Degree: 4 Title: California Demo, Type: StoryMap, Degree: 4 Title: Offline Cloning Draft Test, Type: Web Experience, Degree: 4 Title: California Demo, Type: StoryMap, Degree: 4 Title: California Demo, Type: StoryMap, Degree: 4 Title: California Demo, Type: StoryMap, Degree: 4 Title: Offline Cloning Draft Test, Type: Web Experience, Degree: 4 Title: Offline Cloning Draft Test, Type: Web Experience, Degree: 4 Title: California Demo, Type: StoryMap, Degree: 4 Title: Offline Cloning Draft Test, Type: Web Experience, Degree: 4 Title: SimpleWebMap, Type: Web Map, Degree: 4 Title: Arkansas Hospitals, Type: Web Map, Degree: 4 Title: Nothing Funny About This Test - updated PR, Type: Web Experience, Degree: 3 Title: Hilarious Little Test, Type: Web Experience, Degree: 3 Title: Hilarious Little Test2, Type: Web Experience, Degree: 3 Title: Hilarious Little Test - updated PR, Type: Web Experience, Degree: 3 Title: Hilarious Little Test - updated PR1, Type: Web Experience, Degree: 3 Title: No Image Test, Type: Web Experience, Degree: 3 Title: Nothing Funny About This Test, Type: Web Experience, Degree: 3 Title: No Image Test, Type: Web Experience, Degree: 3 Title: Nothing Funny About This Test, Type: Web Experience, Degree: 3 Title: Esri Offices, Fancy, Type: Web Map, Degree: 3 Title: Esri Offices, Fancy, Type: Web Map, Degree: 3 Title: TerrainMap, Type: Web Map, Degree: 3 Title: TerrainMap, Type: Web Map, Degree: 3 Title: TerrainMap, Type: Web Map, Degree: 3
Finally, we can construct a custom function that can do the two prior bits, plus sort items by type, and define the result count. This allows us to easily filter and search our org's items with additional context on the weight of their dependencies.
def filter_nodes(direction = "out", deep = False, descending = True, item_type = None, result_count: int = None):
if item_type:
nodes_filtered = [node for node in new_graph.all_items() if node.item and node.item.type == item_type]
else:
nodes_filtered = new_graph.all_items()
if direction == "in" and deep:
nodes_weighted = [(node, len(node.required_by())) for node in nodes_filtered]
elif direction == "out" and deep:
nodes_weighted = [(node, len(node.requires())) for node in nodes_filtered]
elif direction == "in" and not deep:
nodes_weighted = [(node, len(node.contained_by())) for node in nodes_filtered]
else:
nodes_weighted = [(node, len(node.contains())) for node in nodes_filtered]
nodes_sorted = sorted(nodes_weighted, key=lambda item: item[1], reverse = descending)
if result_count and result_count > len(nodes_sorted):
result_count = len(nodes_sorted)
for node, degree in nodes_sorted[:result_count]:
if node.item:
print(f"Title: {node.item.title}, Type: {node.item.type}, Degree: {degree}")
else:
print(f"ID: {node.id}, Degree: {degree}")Part 3: Automating this Notebook
Everything we've done in this notebook is purely informative- no changes, additions, or deletions to org items were done. All we've done is update and manipulate or ItemGraph in memory and written out to file. So what does this mean?
This means that you can download this very notebook and run it for yourself. Keeping this notebook run on a consistent timeframe will provide you with an up-to-date graph to analyze your org items. ArcGIS Notebooks, in both ArcGIS Online and ArcGIS Enterprise, provide us with an easy way to schedule automation of notebooks. Let's take a look how.
Scheduling a Task
A "task" is a command to run a given notebook. You can create scheduled tasks that dictate when and how often to run a notebook. Each time the notebook is run according to the schedule you set, the site opens a new container and runs the notebook without user control.
To access this function, open the "Tasks" pane in the notebook editor. You can schedule it to run just once in the future or on a recurring basis— from once every 15 minutes to once every 6 months.
Once you click on the Tasks pane, you will be prompted to provide a title for the Task.
You can also set other specific parameters for when and how often you would like for this notebook to run.
Once you have created this task, you can find it under the Tasks pane, and can pause, delete or edit it as necessary.
Note: For Notebooks in ArcGIS Online, you can have a maximum of 10 active tasks across all your notebooks. If you pause some of your active tasks, you will be allowed to create more tasks until you reach 10 active tasks again. For Notebooks in ArcGIS Enterprise, the default number of maximum active tasks is set at 20. Administrators of an organization have the privilege to change that limit.