Clone Portal users, groups and content¶
This sample notebook can be used for cloning a portal, from say, a staging to a production environment. It clones the users, groups and the content. It does not copy over services though, and works at the tier of portal items.
Note: To use this notebook as a Python script, check out the accompanying SDK GitHub repository. Running this as a script from a Python IDE allows you to set breakpoints, debug, and inspect the script when an exception is raised.
from arcgis.gis import GIS
from IPython.display import display
from getpass import getpass
Define the source and target portals¶
To start with, define the source and target portals. Connect to them using accounts with administrative privileges:
source_password = getpass()
target_password = getpass()
source = GIS("source portal url", username, source_password)
target = GIS("target portal url", username, target_password)
target_admin_username = 'admin'
Users¶
List the users in the source and target portals. We do not want to copy over system accounts since those would be available in the target portal as well. Hence, filter the search by negating any account that starts with 'esri_'. We also do not want to copy over the initial administrator account as one would be present in the target as well. Hence, negate the account that starts with admin which happens to be the administrator account on source portal.
#!esri_ & !admin
source_users = source.users.search('!esri_ & !admin')
for user in source_users:
print(user.username + "\t:\t" + str(user.role))
Get the number of users to migrate:
len(source_users)
Get the list of users already present in the target portal. Similar to earlier, filter out system and initial administrator accounts. The name of the admin account on target portal is admin as well in this example.
# filter out system and initial administrator accounts
target_users = target.users.search('!esri_ & !admin & !system_publisher')
target_users
If users found on source portal were already in the target portal, run the following code to delete them. You can choose to not delete them as well.
Remove existing users from target portal¶
If you want to clean up the target portal except for the initial administrator account, run the cell below. As you delete, you may opt to assign their content to the initial administrator account.
for source_user in source_users:
try:
target_user = target.users.get(source_user.username)
if target_user is not None:
print('Deleting user: ' + target_user.fullName)
target_user.reassign_to(target_admin_username)
target_user.delete()
except:
print('User {} does not exist in Target Portal'.format(source_user.username))
Copy Users¶
Create a function that will accept connection to the target portal, User objects from source portal and password to create users with. In addition to creating the users, this function will set their access, description, tags and other similar properties from source. If a user by the same name already exists in the target portal (possible if you opted not to clean out the target portal) then this function prints out an error message.
def copy_user(target_portal, source_user, password):
# See if the user has firstName and lastName properties
try:
first_name = source_user.firstName
last_name = source_user.lastName
except:
# if not, split the fullName
full_name = source_user.fullName
first_name = full_name.split()[0]
try:
last_name = full_name.split()[1]
except:
last_name = 'NoLastName'
try:
# create user
target_user = target_portal.users.create(source_user.username, password, first_name,
last_name, source_user.email,
source_user.description, source_user.role)
# update user properties
target_user.update(source_user.access, source_user.preferredView,
source_user.description, source_user.tags,
source_user.get_thumbnail_link(),
culture=source_user.culture, region=source_user.region)
return target_user
except Exception as Ex:
print(str(Ex))
print("Unable to create user "+ source_user.username)
return None
For each user in source portal, make a corresponding user in target portal. In this sample, we provide a common password to all users TestPassword@123 as we are creating users off the built-in identity store. If you are creating users off your enterprise identity store, you can ignore the password parameter and use the provider and idp_username parameters as explained in the API reference doc.
for user in source_users:
print("Creating user: " + user.username)
copy_user(target, user, 'TestPassword@123')
Verify that users have been added to target portal:
target_users = target.users.search()
target_users
Thus, users have been successfully added to the target portal
Groups¶
List the groups in the source and target portals. Similar to how we searched for users, we will ignore the system created and default groups as they would be available on the target portal as well.
# filter out system created groups
source_groups = source.groups.search("!owner:esri_* & !Basemaps")
source_groups
target_groups = target.groups.search("!owner:esri_* & !Basemaps")
target_groups
If any of the groups from source are already in the target, run the following code to delete them. If the group belongs to any of default user accounts, don't delete it. This step is optional, you may choose to not delete those groups if you prefer to retain them as is.
for tg in target_groups:
for sg in source_groups:
if sg.title == tg.title and (not tg.owner.startswith('esri_')):
print("Cleaning up group {} in target Portal...".format(tg.title))
tg.delete()
break
Copy Groups¶
Let us create a function that will clone the groups one at a time. As you call this function in a loop for each group, it reads the source group's properties, downloads thumbnail into a temporary file then creates a similar named group on target and applies those properties and thumbnail. If one of your portals is an organization on ArcGIS Online and other is an ArcGIS Enterprise, certain privacy properties need to be adapted. This function takes care of that. After creating the group, it finds which users were members of it and adds them appropriately.
import tempfile
GROUP_COPY_PROPERTIES = ['title', 'description', 'tags', 'snippet', 'phone',
'access', 'isInvitationOnly']
def copy_group(target, source, source_group):
with tempfile.TemporaryDirectory() as temp_dir:
try:
target_group = {}
for property_name in GROUP_COPY_PROPERTIES:
target_group[property_name] = source_group[property_name]
if source_group['access'] == 'org' and target.properties['portalMode'] == 'singletenant':
#cloning from ArcGIS Online to ArcGIS Enterprise
target_group['access'] = 'public'
elif source_group['access'] == 'public'\
and source.properties['portalMode'] == 'singletenant'\
and target.properties['portalMode'] == 'multitenant'\
and 'id' in target.properties:
#cloning from ArcGIS Enterprise to ArcGIS Online org
target_group['access'] = 'org'
# Download the thumbnail (if one exists)
thumbnail_file = None
if 'thumbnail' in group:
target_group['thumbnail'] = group.download_thumbnail(temp_dir)
# Create the group in the target portal
copied_group = target.groups.create_from_dict(target_group)
# Reassign all groups to correct owners, add users, and find shared items
members = group.get_members()
if not members['owner'] == target_admin_username:
copied_group.reassign_to(members['owner'])
if members['users']:
copied_group.add_users(members['users'])
return copied_group
except:
print("Error creating " + source_group['title'])
For each group in source portal, make a corresponding group in target portal.
from IPython.display import display
for group in source_groups:
target_group = copy_group(target, source, group)
if target_group:
display(target_group)
As you can see, we were able to add the groups with their thumbnails. Now let us verify that groups can be listed on the target portal:
target_groups = target.groups.search()
target_groups
With this part of the sample, we have successfully created users, groups and added the appropriate users to these groups. Thus, you can call the get_members() method one of the groups to view its members:
group1 = target_groups[0]
group1.get_members()
Items¶
Copying items consists of multiple steps as explained in the following section of the sample:
For each user create a mapping of itemId to the Item¶
Do this for every folder in the user's account on the source portal
source_items_by_id = {}
for user in source_users:
num_items = 0
num_folders = 0
print("Collecting item ids for {}".format(user.username), end="\t\t")
user_content = user.items()
# Get item ids from root folder first
for item in user_content:
num_items += 1
source_items_by_id[item.itemid] = item
# Get item ids from each of the folders next
folders = user.folders
for folder in folders:
num_folders += 1
folder_items = user.items(folder=folder['title'])
for item in folder_items:
num_items += 1
source_items_by_id[item.itemid] = item
print("Number of folders {} # Number of items {}".format(str(num_folders), str(num_items)))
Let us print the dictionary of {item_id : Item object}
source_items_by_id
Prepare sharing information for each item¶
Using the dictionary we created above, find to which groups are each of the items shared to.
for group in source_groups:
#iterate through each item shared to the source group
for group_item in group.content():
try:
#get the item
item = source_items_by_id[group_item.itemid]
if item is not None:
if not 'groups'in item:
item['groups'] = []
#assign the target portal's corresponding group's name
item['groups'].append(group['title'])
except:
print("Cannot find item : " + group_item.itemid)
Print a mapping of item and its group membership¶
for key in source_items_by_id.keys():
item = source_items_by_id[key]
print("\n{:40s}".format(item.title), end = " # ")
if 'groups' in item:
print(item.access, end = " # ")
print(item.groups, end = "")
As we can see from above, some items are shared to a few groups while some are not.
Copy Items¶
Below we define a function that you can call in a loop for each item in the dictionary we composed earlier. If the item is a text based item such as a Web Map or a file based item such as a layer package, it downloads the item's data to a temporary directory and uses that for creating the target item during cloning. You can find the exhaustive list of different items that you can upload to your portal and their corresponding item types from the REST API documentation. For brevity, this sample covers only a subset of those items. Note, if the item points to a web layer URL, the target item would also point to the same URL.
TEXT_BASED_ITEM_TYPES = frozenset(['Web Map', 'Feature Service', 'Map Service','Web Scene',
'Image Service', 'Feature Collection',
'Feature Collection Template',
'Web Mapping Application', 'Mobile Application',
'Symbol Set', 'Color Set',
'Windows Viewer Configuration'])
FILE_BASED_ITEM_TYPES = frozenset(['File Geodatabase','CSV', 'Image', 'KML', 'Locator Package',
'Map Document', 'Shapefile', 'Microsoft Word', 'PDF',
'Microsoft Powerpoint', 'Microsoft Excel', 'Layer Package',
'Mobile Map Package', 'Geoprocessing Package', 'Scene Package',
'Tile Package', 'Vector Tile Package'])
ITEM_COPY_PROPERTIES = ['title', 'type', 'typeKeywords', 'description', 'tags',
'snippet', 'extent', 'spatialReference', 'name',
'accessInformation', 'licenseInfo', 'culture', 'url']
We define the copy function for items below. This function gets the properties of the item from source and applies it to the target. If the items were saved inside a folder, it creates that folder on the target as well. Finally, it sets the privacy (sharing) properties similar to how it was on the source portal.
def copy_item(target, source_item):
try:
with tempfile.TemporaryDirectory() as temp_dir:
item_properties = {}
for property_name in ITEM_COPY_PROPERTIES:
item_properties[property_name] = source_item[property_name]
data_file = None
if source_item.type in TEXT_BASED_ITEM_TYPES:
# If its a text-based item, then read the text and add it to the request.
text = source_item.get_data(False)
item_properties['text'] = text
elif source_item.type in FILE_BASED_ITEM_TYPES:
# download data and add to the request as a file
data_file = source_item.download(temp_dir)
thumbnail_file = source_item.download_thumbnail(temp_dir)
metadata_file = source_item.download_metadata(temp_dir)
#find item's owner
source_item_owner = source.users.search(source_item.owner)[0]
#find item's folder
item_folder_titles = [f['title'] for f in source_item_owner.folders
if f['id'] == source_item.ownerFolder]
folder_name = None
if len(item_folder_titles) > 0:
folder_name = item_folder_titles[0]
#if folder does not exist for target user, create it
if folder_name:
target_user = target.users.search(source_item.owner)[0]
target_user_folders = [f['title'] for f in target_user.folders
if f['title'] == folder_name]
if len(target_user_folders) == 0:
#create the folder
target.content.create_folder(folder_name, source_item.owner)
# Add the item to the target portal, assign owner and folder
target_item = target.content.add(item_properties, data_file, thumbnail_file,
metadata_file, source_item.owner, folder_name)
#Set sharing (privacy) information
share_everyone = source_item.access == 'public'
share_org = source_item.access in ['org', 'public']
share_groups = []
if source_item.access == 'shared':
share_groups = source_item.groups
target_item.share(share_everyone, share_org, share_groups)
return target_item
except Exception as copy_ex:
print("\tError copying " + source_item.title)
print("\t" + str(copy_ex))
return None
Copy over each item. While doing so, construct a dictionary mapping of source item's ID with target item's ID
source_target_itemId_map = {}
for key in source_items_by_id.keys():
source_item = source_items_by_id[key]
print("Copying {} \tfor\t {}".format(source_item.title, source_item.owner))
target_item = copy_item(target, source_item)
if target_item:
source_target_itemId_map[key] = target_item.itemid
else:
source_target_itemId_map[key] = None
We have successfully cloned all the items from source to target. We can query the contents of one of the users below to verify:
user1 = target.users.search()[2]
user1
user1.items()
We could query the folders belonging to this user and the items within as well
user1.folders
user1.items(folder=user1.folders[0]['title'])
Establish relationship between items¶
So far, we have successfully cloned users, groups and items from source to target. Next, we will establish identical relationships between items as they were in the source portal.
RELATIONSHIP_TYPES = frozenset(['Map2Service', 'WMA2Code',
'Map2FeatureCollection', 'MobileApp2Code', 'Service2Data',
'Service2Service'])
Below, we loop through each item in source portal, find to which other item it is related and the type of that relationship. If a relationship is found, we find the corresponding items in target and establish the same relationship. To make this work, we will make use of the dictionary that maps the itemIds on source and target we created during the item clone stage. Let us take a look at that dictionary below:
source_target_itemId_map
for key in source_target_itemId_map.keys():
source_item = source_items_by_id[key]
target_itemid = source_target_itemId_map[key]
target_item = target.content.get(target_itemid)
print(source_item.title + " # " + source_item.type)
for relationship in RELATIONSHIP_TYPES:
try:
source_related_items = source_item.related_items(relationship)
for source_related_item in source_related_items:
print("\t\t" + source_related_item.title + " # " +
source_related_item.type +"\t## " + relationship)
#establish same relationship amongst target items
print("\t\t" + "establishing relationship in target portal", end=" ")
target_related_itemid = source_target_itemId_map[source_related_item.itemid]
target_related_item = target.content.get(target_related_itemid)
status = target_item.add_relationship(target_related_item, relationship)
print(str(status))
except Exception as rel_ex:
print("\t\t Error when checking for " + relationship + " : " + str(rel_ex))
continue
Conclusion¶
Thus, with this notebook, we have successfully cloned groups, users and their contents. Note, this notebook did not copy over the services that power the service based items. Such items continue to point to the same URL as the ones in source portal did. As long as those URLs remain accessible, the web maps and layer items continue to be usable.
To run this notebook as a Python script, checkout the Python scripts in the accompanying SDK GitHub repository.
Feedback on this topic?