Clone Portal users, groups and content¶

This sample notebook can be used for cloning a portal, from say, a staging to a production environment. It clones the users, groups and the content. It does not copy over services though, and works at the tier of portal items.

Note: To user this notebook as a Python script, checkout the accompanying SDK GitHub repository. Running this as a script from a Python IDE allows you to set breakpoints, debug and inspect the script when an exception is raised.

In [1]:

from arcgis.gis import GIS
from IPython.display import display
from getpass import getpass

Define the source and target portals¶

To start with, define the source and target portals. Connect to them using accounts with administrative privileges:

In [2]:

source_password = getpass()
target_password = getpass()
source = GIS("source portal url", username, source_password)
target = GIS("target portal url", username, target_password)
target_admin_username = 'admin'

········
········

Users¶

List the users in the source and target portals. We do not want to copy over system accounts since those would be available in the target portal as well. Hence, filter the search by negating any account that starts with 'esri_'. We also do not want to copy over the initial administrator account as one would be present in the target as well. Hence, negate the account that starts with admin which happens to be the administrator account on source portal.

In [3]:

#!esri_ & !admin
source_users = source.users.search('!esri_ & !admin')
for user in source_users:
    print(user.username + "\t:\t" + str(user.role))

brown.rogers	:	org_user
davis.reed	:	org_admin
johnson.stewart	:	org_user
jones.morris	:	org_user
miller.cook	:	org_publisher
moore.bell	:	org_publisher
project_archiver	:	org_user
smith.collins	:	org_admin
taylor.murphy	:	org_publisher
williams.sanchez	:	org_user
wilson.morgan	:	org_publisher

Get the number of users to migrate:

In [4]:

len(source_users)

Out[4]:

Get the list of users already present in the target portal. Similar to earlier, filter out system and initial administrator accounts. The name of the admin account on target portal is admin as well in this example.

In [5]:

# filter out system and initial administrator accounts
target_users = target.users.search('!esri_ & !admin & !system_publisher')
target_users

Out[5]:

[<User username:arcgis_python_api>, <User username:publisher1>]

If users found on source portal were already in the target portal, run the following code to delete them. You can choose to not delete them as well.

Remove existing users from target portal¶

If you want to clean up the target portal except for the initial administrator account, run the cell below. As you delete, you may opt to assign their content to the initial administrator account.

In [6]:

for source_user in source_users:
    try:
        target_user = target.users.get(source_user.username)
        if target_user is not None:
            print('Deleting user: ' + target_user.fullName)
            target_user.reassign_to(target_admin_username)
            target_user.delete()
    except:
        print('User {} does not exist in Target Portal'.format(source_user.username))

Copy Users¶

Create a function that will accept connection to the target portal, User objects from source portal and password to create users with. In addition to creating the users, this function will set their access, description, tags and other similar properties from source. If a user by the same name already exists in the target portal (possible if you opted not to clean out the target portal) then this function prints out an error message.

In [7]:

def copy_user(target_portal, source_user, password):
    # See if the user has firstName and lastName properties
    try:
        first_name = source_user.firstName
        last_name = source_user.lastName
    except:
        # if not, split the fullName
        full_name = source_user.fullName
        first_name = full_name.split()[0]
        try:
            last_name = full_name.split()[1]
        except:
            last_name = 'NoLastName'

    try:
        # create user
        target_user = target_portal.users.create(source_user.username, password, first_name, 
                                                 last_name, source_user.email, 
                                                 source_user.description, source_user.role)

        # update user properties
        target_user.update(source_user.access, source_user.preferredView,
                           source_user.description, source_user.tags, 
                           source_user.get_thumbnail_link(),
                           culture=source_user.culture, region=source_user.region)
        return target_user
    
    except Exception as Ex:
        print(str(Ex))
        print("Unable to create user "+ source_user.username)
        return None

For each user in source portal, make a corresponding user in target portal. In this sample, we provide a common password to all users TestPassword@123 as we are creating users off the built-in identity store. If you are creating users off your enterprise identity store, you can ignore the password parameter and use the provider and idp_username parameters as explained in the API reference doc.

In [8]:

for user in source_users:
    print("Creating user: " + user.username)
    copy_user(target, user, 'TestPassword@123')

Creating user: brown.rogers
Creating user: davis.reed
Creating user: johnson.stewart
Creating user: jones.morris
Creating user: miller.cook
Creating user: moore.bell
Creating user: project_archiver
Creating user: smith.collins
Creating user: taylor.murphy
Creating user: williams.sanchez
Creating user: wilson.morgan

Verify that users have been added to target portal:

In [9]:

target_users = target.users.search()
target_users

Out[9]:

[<User username:admin>,
 <User username:arcgis_python_api>,
 <User username:brown.rogers>,
 <User username:davis.reed>,
 <User username:esri_boundaries>,
 <User username:esri_demographics>,
 <User username:esri_livingatlas>,
 <User username:esri_nav>,
 <User username:johnson.stewart>,
 <User username:jones.morris>,
 <User username:miller.cook>,
 <User username:moore.bell>,
 <User username:project_archiver>,
 <User username:publisher1>,
 <User username:smith.collins>,
 <User username:system_publisher>,
 <User username:taylor.murphy>,
 <User username:williams.sanchez>,
 <User username:wilson.morgan>]

Thus, users have been successfully added to the target portal

Groups¶

List the groups in the source and target portals. Similar to how we searched for users, we will ignore the system created and default groups as they would be available on the target portal as well.

In [10]:

# filter out system created groups
source_groups = source.groups.search("!owner:esri_* & !Basemaps")
source_groups

Out[10]:

[<Group title:"Central Services" owner:admin>,
 <Group title:"Compliance" owner:admin>,
 <Group title:"Customer Service, Finance, Billing and Accounting" owner:admin>,
 <Group title:"Demographic Content" owner:admin>]

In [11]:

target_groups = target.groups.search("!owner:esri_* & !Basemaps")
target_groups

Out[11]:

[<Group title:"Featured Maps and Apps" owner:admin>]

If any of the groups from source are already in the target, run the following code to delete them. If the group belongs to any of default user accounts, don't delete it. This step is optional, you may choose to not delete those groups if you prefer to retain them as is.

In [12]:

for tg in target_groups:
    for sg in source_groups:
        if sg.title == tg.title and (not tg.owner.startswith('esri_')):
            print("Cleaning up group {} in target Portal...".format(tg.title))
            tg.delete()
            break

Copy Groups¶

Let us create a function that will clone the groups one at a time. As you call this function in a loop for each group, it reads the source group's properties, downloads thumbnail into a temporary file then creates a similar named group on target and applies those properties and thumbnail. If one of your portals is an organization on ArcGIS Online and other is an ArcGIS Enterprise, certain privacy properties need to be adapted. This function takes care of that. After creating the group, it finds which users were members of it and adds them appropriately.

In [13]:

import tempfile

GROUP_COPY_PROPERTIES = ['title', 'description', 'tags', 'snippet', 'phone',
                         'access', 'isInvitationOnly']

def copy_group(target, source, source_group):
    
    with tempfile.TemporaryDirectory() as temp_dir:
        try:
            target_group = {}

            for property_name in GROUP_COPY_PROPERTIES:
                target_group[property_name] = source_group[property_name]

            if source_group['access'] == 'org' and target.properties['portalMode'] == 'singletenant':
                #cloning from ArcGIS Online to ArcGIS Enterprise
                target_group['access'] = 'public'

            elif source_group['access'] == 'public'\
                 and source.properties['portalMode'] == 'singletenant'\
                 and target.properties['portalMode'] == 'multitenant'\
                 and 'id' in target.properties:
                    #cloning from ArcGIS Enterprise to ArcGIS Online org
                    target_group['access'] = 'org'

            # Download the thumbnail (if one exists)
            thumbnail_file = None
            if 'thumbnail' in group:
                target_group['thumbnail'] = group.download_thumbnail(temp_dir)

            # Create the group in the target portal
            copied_group = target.groups.create_from_dict(target_group)

            # Reassign all groups to correct owners, add users, and find shared items
            members = group.get_members()
            if not members['owner'] == target_admin_username:
                copied_group.reassign_to(members['owner'])
            if members['users']:
                copied_group.add_users(members['users'])
            return copied_group
        except:
            print("Error creating " + source_group['title'])

For each group in source portal, make a corresponding group in target portal.

In [14]:

from IPython.display import display
for group in source_groups:
    target_group = copy_group(target, source, group)
    if target_group:
        display(target_group)

Central Services

Summary: The authoritative service catalog.
Description: This Group contains an inventory of map services for our organization. These map services serve as building blocks for all maps and apps throughout the organization.
Owner: admin
Created: April 04, 2017

Compliance

Summary: Regulatory compliance tracking & reporting.
Description: A group dealing with government and industry association regulatory compliance and reporting.
Owner: admin
Created: April 04, 2017

Customer Service, Finance, Billing and Accounting

Summary: The Water & Sewer Billing and Collection Division manage the water and sewer accounts for residents.
Description: Typical types of users and roles that part of this group:Human Resources/PayrollAdministrative Services Officer I, IIAdministrative AideAccountantAccount ClerkPayroll ClerkDepartment AnalystRisk Management Analyst I,II,IIIAccounting TechnicianEngineering TechnicianCommon Task or Responsibilities of the users in this group:Customer ServiceReal Estate ServicesBudgets/Accounting/Long Range Financial PlanningSafety/Risk ManagementFinancingRecords Management/Clerical ServicesAuditingBill Collection and paymentConnections/DisconnectsReduced Rates / Leak AdjustmentsPool AdjustmentIrrigation and Hydrant MetersEmployee Development
Owner: admin
Created: April 04, 2017

Demographic Content

Summary: Esri demographic data with national coverage
Description: A catalog of Esri provided demographic content for use at the utility.
Owner: admin
Created: April 04, 2017

As you can see, we were able to add the groups with their thumbnails. Now let us verify that groups can be listed on the target portal:

In [15]:

target_groups = target.groups.search()
target_groups

Out[15]:

[<Group title:"Central Services" owner:admin>,
 <Group title:"Compliance" owner:admin>,
 <Group title:"Customer Service, Finance, Billing and Accounting" owner:admin>,
 <Group title:"Demographic Content" owner:admin>,
 <Group title:"Esri Boundary Layers" owner:esri_boundaries>,
 <Group title:"Esri Demographic Layers" owner:esri_demographics>,
 <Group title:"Featured Maps and Apps" owner:admin>,
 <Group title:"Living Atlas" owner:esri_livingatlas>,
 <Group title:"Living Atlas Analysis Layers" owner:esri_livingatlas>,
 <Group title:"Navigator Maps" owner:esri_nav>]

With this part of the sample, we have successfully created users, groups and added the appropriate users to these groups. Thus, you can call the get_members() method one of the groups to view its members:

In [17]:

group1 = target_groups[0]
group1.get_members()

Out[17]:

{'admins': ['admin'],
 'owner': 'admin',
 'users': ['brown.rogers',
  'johnson.stewart',
  'taylor.murphy',
  'smith.collins']}

Items¶

Copying items consists of multiple steps as explained in the following section of the sample:

For each user create a mapping of itemId to the Item
Prepare sharing information for each item
1. Print a mapping of item and its group membership
Copy items one by one
Establish relationship between items

For each user create a mapping of itemId to the `Item`¶

Do this for every folder in the user's account on the source portal

In [18]:

source_items_by_id = {}
for user in source_users:
    num_items = 0
    num_folders = 0
    print("Collecting item ids for {}".format(user.username), end="\t\t")
    user_content = user.items()
    
    # Get item ids from root folder first
    for item in user_content:
        num_items += 1
        source_items_by_id[item.itemid] = item 
    
    # Get item ids from each of the folders next
    folders = user.folders
    for folder in folders:
        num_folders += 1
        folder_items = user.items(folder=folder['title'])
        for item in folder_items:
            num_items += 1
            source_items_by_id[item.itemid] = item
    
    print("Number of folders {} # Number of items {}".format(str(num_folders), str(num_items)))

Collecting item ids for brown.rogers		Number of folders 1 # Number of items 3
Collecting item ids for davis.reed		Number of folders 1 # Number of items 3
Collecting item ids for johnson.stewart		Number of folders 1 # Number of items 3
Collecting item ids for jones.morris		Number of folders 1 # Number of items 3
Collecting item ids for miller.cook		Number of folders 1 # Number of items 3
Collecting item ids for moore.bell		Number of folders 1 # Number of items 3
Collecting item ids for project_archiver		Number of folders 7 # Number of items 18
Collecting item ids for smith.collins		Number of folders 1 # Number of items 4
Collecting item ids for taylor.murphy		Number of folders 1 # Number of items 3
Collecting item ids for williams.sanchez		Number of folders 1 # Number of items 3
Collecting item ids for wilson.morgan		Number of folders 1 # Number of items 3

Let us print the dictionary of {item_id : Item object}

In [19]:

source_items_by_id

Out[19]:

{'0c8ffe4ab7754eedbd2c514032f7e913': <Item title:"IN" type:Feature Service owner:williams.sanchez>,
 '0fdcdd6eed9e4e6aa83575da5c4d8ff0': <Item title:"IN" type:CSV owner:williams.sanchez>,
 '10eff50fda644f2fa9261ee925a24495': <Item title:"set2_empty" type:Map Document owner:project_archiver>,
 '14935a49a8544d8eb0eeadaa6216bbba': <Item title:"set2_USAcities" type:File Geodatabase owner:project_archiver>,
 '1506548f309140d89a19886913d8395a': <Item title:"Miller Cook response locations" type:Web Map owner:miller.cook>,
 '1cb04f79f8a74826bab704efe5206603': <Item title:"set1_gov_sites_registration" type:Microsoft Excel owner:project_archiver>,
 '1d8820ef73064498983af1af326ddb9d': <Item title:"Johnson Stewart response locations" type:Web Map owner:johnson.stewart>,
 '28984d7a69f34b51b07a087800931d6f': <Item title:"Jones Morris response locations" type:Web Map owner:jones.morris>,
 '297a1819ba9b43ffb8adcab6a7eebd53': <Item title:"LA" type:CSV owner:taylor.murphy>,
 '464fdece247e46b1ba861a7ac92701a4': <Item title:"FL" type:CSV owner:davis.reed>,
 '48f17e7559ca4d65b5eab5b78cabf085': <Item title:"set1_fortune500" type:File Geodatabase owner:project_archiver>,
 '4a6c6b94d1d840f3a19d7da922be4c82': <Item title:"KS" type:CSV owner:smith.collins>,
 '54cf94db12e64092bef153b9f2c07176': <Item title:"set2_australia" type:GeoJson owner:project_archiver>,
 '56d1921b6cfb46e89bd697668cceb708': <Item title:"set1_mapping_tech" type:Microsoft Powerpoint owner:project_archiver>,
 '59132371ac22462a899bcaf7b561f1cd': <Item title:"set2_Voronoi-diagram" type:Microsoft Word owner:project_archiver>,
 '67503e5077f240738e091e4c1da13745': <Item title:"set3_Streets" type:Map Document owner:project_archiver>,
 '715f8038c23b4b53a1280ae1b7b4a9bc': <Item title:"Williams Sanchez response locations" type:Web Map owner:williams.sanchez>,
 '71b66b8e628b46c9850cefd91f67f604': <Item title:"NV" type:Feature Service owner:johnson.stewart>,
 '73118cf2a8334a9884f269f81ab25c45': <Item title:"Taylor Murphy response locations" type:Web Map owner:taylor.murphy>,
 '734529a8daad4317bc6daedfe01fdc9f': <Item title:"set2_catalina-points" type:KML owner:project_archiver>,
 '75c63c8de25943f9b5ed9c477e556034': <Item title:"set1_Chicago" type:CSV owner:project_archiver>,
 '7be7710b5e4b44cf836139a115825b2e': <Item title:"ID" type:CSV owner:wilson.morgan>,
 '7c4fdc5508004253960993bbab02bf61': <Item title:"NC" type:CSV owner:jones.morris>,
 '7e854deebcd1442eb850a3ea84732679': <Item title:"set1_major_cities" type:Locator Package owner:project_archiver>,
 '819d0cbed11f4f81a98e7f388fb8eeab': <Item title:"Smith Collins response locations" type:Web Map owner:smith.collins>,
 '882009025e11471caad3f48389fde221': <Item title:"FL" type:Feature Service owner:davis.reed>,
 '8a6b1ed1b7be4cdeb298020cf5108c70': <Item title:"set2_Chicago" type:CSV owner:project_archiver>,
 '8d025e7368974649a3eb0691656dfa57': <Item title:"set2_SD_crime" type:Map Document owner:project_archiver>,
 '97d15ea9057549cdbe67344fe129c758': <Item title:"ID" type:Feature Service owner:wilson.morgan>,
 '9afed1a827914a53af34af490f64ecdf': <Item title:"Moore Bell response locations" type:Web Map owner:moore.bell>,
 'a0929a38db3240cfaffbc254d00e1827': <Item title:"AZ" type:Feature Service owner:moore.bell>,
 'a89398de64444d58a9ea6a0cc407c6b7': <Item title:"LA" type:Feature Service owner:taylor.murphy>,
 'aaa6ad38f9c24b4294a4c6d0994fc7c7': <Item title:"AR" type:Feature Service owner:brown.rogers>,
 'b18c614bc00d40d0a216754d1aa8b150': <Item title:"Brown Rogers response locations" type:Web Map owner:brown.rogers>,
 'b8c648204c804c289930b9e121dd8fdb': <Item title:"AZ" type:CSV owner:moore.bell>,
 'ba3d9b3e6793406696497be34607de44': <Item title:"set1_GeoJson" type:PDF owner:project_archiver>,
 'c56e60ab29f1400fbc46cc2732724ef8': <Item title:"Wilson Morgan response locations" type:Web Map owner:wilson.morgan>,
 'd65f34a670654818a010771d9a896760': <Item title:"Davis Reed response locations" type:Web Map owner:davis.reed>,
 'd6818b3d16c847b5ab9986d708b992a2': <Item title:"set2_counties" type:Locator Package owner:project_archiver>,
 'd8fdaa873f724c7abdca62a42de0c97a': <Item title:"USA_cities_Fortune_500" type:Map Document owner:project_archiver>,
 'dee8ae7fede64f8e94c39d4f6aff8d6f': <Item title:"KS" type:Feature Service owner:smith.collins>,
 'e0e716ac1d774cb190765dd7a2a7421f': <Item title:"set1_india" type:GeoJson owner:project_archiver>,
 'e76f3fab66534499bdc394558a645357': <Item title:"NV" type:CSV owner:johnson.stewart>,
 'ea36c5dbd4e142079c43b17cb49b4258': <Item title:"set1_GeoJson" type:Microsoft Word owner:project_archiver>,
 'ebd57622fb984f1c96122c4b442ea6d7': <Item title:"AR" type:CSV owner:brown.rogers>,
 'edba588da56b4a3ab2f2df576f9c6b99': <Item title:"NH" type:CSV owner:miller.cook>,
 'eed82e9d95dc4ad6a0f3d21e530e0a8f': <Item title:"Smith Collins response locations" type:Web Map owner:smith.collins>,
 'f3f87cb385ea4ae291fd6557338e8e0b': <Item title:"NH" type:Feature Service owner:miller.cook>,
 'f9866c5a2be0428abe1c7097993d9b36': <Item title:"NC" type:Feature Service owner:jones.morris>}

Using the dictionary we created above, find to which groups are each of the items shared to.

In [20]:

for group in source_groups:
    #iterate through each item shared to the source group
    for group_item in group.content():
        try:
            #get the item
            item = source_items_by_id[group_item.itemid]
            if item is not None:
                if not 'groups'in item:
                    item['groups'] = []
                
                #assign the target portal's corresponding group's name
                item['groups'].append(group['title'])
        except:
            print("Cannot find item : " + group_item.itemid)

Print a mapping of item and its group membership¶

In [21]:

for key in source_items_by_id.keys():
    item = source_items_by_id[key]
    print("\n{:40s}".format(item.title), end = " # ")
    if 'groups' in item:
        print(item.access, end = " # ")
        print(item.groups, end = "")

KS                                       # 
NC                                       # 
AR                                       # 
set2_catalina-points                     # 
FL                                       # 
KS                                       # 
set1_GeoJson                             # 
set2_australia                           # 
NV                                       # 
AZ                                       # 
NV                                       # 
FL                                       # 
set3_Streets                             # 
set2_counties                            # 
ID                                       # 
Brown Rogers response locations          # shared # ['Central Services']
set1_Chicago                             # 
set2_USAcities                           # 
Jones Morris response locations          # shared # ['Customer Service, Finance, Billing and Accounting']
set2_Chicago                             # 
Miller Cook response locations           # shared # ['Demographic Content']
ID                                       # 
set1_fortune500                          # 
set1_gov_sites_registration              # 
Smith Collins response locations         # shared # ['Central Services']
set1_india                               # 
Johnson Stewart response locations       # shared # ['Central Services']
set2_SD_crime                            # 
IN                                       # 
set1_GeoJson                             # 
LA                                       # 
Moore Bell response locations            # shared # ['Compliance', 'Demographic Content']
set2_empty                               # 
Williams Sanchez response locations      # shared # ['Customer Service, Finance, Billing and Accounting']
NH                                       # 
IN                                       # 
AR                                       # 
AZ                                       # 
Wilson Morgan response locations         # shared # ['Compliance', 'Demographic Content']
Davis Reed response locations            # shared # ['Demographic Content']
Smith Collins response locations         # 
NC                                       # 
set1_mapping_tech                        # 
USA_cities_Fortune_500                   # 
Taylor Murphy response locations         # shared # ['Central Services', 'Compliance']
set2_Voronoi-diagram                     # 
set1_major_cities                        # 
LA                                       # 
NH                                       #

As we can see from above, some items are shared to a few groups while some are not.

Copy Items¶

Below we define a function that you can call in a loop for each item in the dictionary we composed earlier. If the item is a text based item such as a Web Map or a file based item such as a layer package, it downloads the item's data to a temporary directory and uses that for creating the target item during cloning. You can find the exhaustive list of different items that you can upload to your portal and their corresponding item types from the REST API documentation. For brevity, this sample covers only a subset of those items. Note, if the item points to a web layer URL, the target item would also point to the same URL.

In [23]:

TEXT_BASED_ITEM_TYPES = frozenset(['Web Map', 'Feature Service', 'Map Service','Web Scene',
                                   'Image Service', 'Feature Collection', 
                                   'Feature Collection Template',
                                   'Web Mapping Application', 'Mobile Application', 
                                   'Symbol Set', 'Color Set',
                                   'Windows Viewer Configuration'])

FILE_BASED_ITEM_TYPES = frozenset(['File Geodatabase','CSV', 'Image', 'KML', 'Locator Package',
                                  'Map Document', 'Shapefile', 'Microsoft Word', 'PDF',
                                  'Microsoft Powerpoint', 'Microsoft Excel', 'Layer Package',
                                  'Mobile Map Package', 'Geoprocessing Package', 'Scene Package',
                                  'Tile Package', 'Vector Tile Package'])

ITEM_COPY_PROPERTIES = ['title', 'type', 'typeKeywords', 'description', 'tags',
                        'snippet', 'extent', 'spatialReference', 'name',
                        'accessInformation', 'licenseInfo', 'culture', 'url']

We define the copy function for items below. This function gets the properties of the item from source and applies it to the target. If the items were saved inside a folder, it creates that folder on the target as well. Finally, it sets the privacy (sharing) properties similar to how it was on the source portal.

In [24]:

def copy_item(target, source_item):
    try:
        with tempfile.TemporaryDirectory() as temp_dir:
            item_properties = {}
            for property_name in ITEM_COPY_PROPERTIES:
                item_properties[property_name] = source_item[property_name]

            data_file = None
            
            if source_item.type in TEXT_BASED_ITEM_TYPES:
                # If its a text-based item, then read the text and add it to the request.
                text = source_item.get_data(False)
                item_properties['text'] = text
            
            elif source_item.type in FILE_BASED_ITEM_TYPES:
                # download data and add to the request as a file
                data_file = source_item.download(temp_dir)

            thumbnail_file = source_item.download_thumbnail(temp_dir)
            metadata_file = source_item.download_metadata(temp_dir)

            #find item's owner
            source_item_owner = source.users.search(source_item.owner)[0]
            
            #find item's folder
            item_folder_titles = [f['title'] for f in source_item_owner.folders 
                                  if f['id'] == source_item.ownerFolder]
            folder_name = None
            if len(item_folder_titles) > 0:
                folder_name = item_folder_titles[0]

            #if folder does not exist for target user, create it
            if folder_name:
                target_user = target.users.search(source_item.owner)[0]
                target_user_folders = [f['title'] for f in target_user.folders
                                       if f['title'] == folder_name]
                if len(target_user_folders) == 0:
                    #create the folder
                    target.content.create_folder(folder_name, source_item.owner)
            
            # Add the item to the target portal, assign owner and folder
            target_item = target.content.add(item_properties, data_file, thumbnail_file, 
                                             metadata_file, source_item.owner, folder_name)
            
            #Set sharing (privacy) information
            share_everyone = source_item.access == 'public'
            share_org = source_item.access in ['org', 'public']
            share_groups = []
            if source_item.access == 'shared':
                share_groups = source_item.groups
            
            target_item.share(share_everyone, share_org, share_groups)
            
            return target_item
        
    except Exception as copy_ex:
        print("\tError copying " + source_item.title)
        print("\t" + str(copy_ex))
        return None

Copy over each item. While doing so, construct a dictionary mapping of source item's ID with target item's ID

In [25]:

source_target_itemId_map = {}
for key in source_items_by_id.keys():
    source_item = source_items_by_id[key]

    print("Copying {} \tfor\t {}".format(source_item.title, source_item.owner))
    target_item = copy_item(target, source_item)
    if target_item:
        source_target_itemId_map[key] = target_item.itemid
    else:
        source_target_itemId_map[key] = None

Copying KS 	for	 smith.collins
Copying NC 	for	 jones.morris
Copying AR 	for	 brown.rogers
Copying set2_catalina-points 	for	 project_archiver
Copying FL 	for	 davis.reed
Copying KS 	for	 smith.collins
Copying set1_GeoJson 	for	 project_archiver
Copying set2_australia 	for	 project_archiver
Copying NV 	for	 johnson.stewart
Copying AZ 	for	 moore.bell
Copying NV 	for	 johnson.stewart
Copying FL 	for	 davis.reed
Copying set3_Streets 	for	 project_archiver
Copying set2_counties 	for	 project_archiver
Copying ID 	for	 wilson.morgan
Copying Brown Rogers response locations 	for	 brown.rogers
Copying set1_Chicago 	for	 project_archiver
Copying set2_USAcities 	for	 project_archiver
Copying Jones Morris response locations 	for	 jones.morris
Copying set2_Chicago 	for	 project_archiver
Copying Miller Cook response locations 	for	 miller.cook
Copying ID 	for	 wilson.morgan
Copying set1_fortune500 	for	 project_archiver
Copying set1_gov_sites_registration 	for	 project_archiver
Copying Smith Collins response locations 	for	 smith.collins
Copying set1_india 	for	 project_archiver
Copying Johnson Stewart response locations 	for	 johnson.stewart
Copying set2_SD_crime 	for	 project_archiver
Copying IN 	for	 williams.sanchez
Copying set1_GeoJson 	for	 project_archiver
Copying LA 	for	 taylor.murphy
Copying Moore Bell response locations 	for	 moore.bell
Copying set2_empty 	for	 project_archiver
Copying Williams Sanchez response locations 	for	 williams.sanchez
Copying NH 	for	 miller.cook
Copying IN 	for	 williams.sanchez
Copying AR 	for	 brown.rogers
Copying AZ 	for	 moore.bell
Copying Wilson Morgan response locations 	for	 wilson.morgan
Copying Davis Reed response locations 	for	 davis.reed
Copying Smith Collins response locations 	for	 smith.collins
Copying NC 	for	 jones.morris
Copying set1_mapping_tech 	for	 project_archiver
Copying USA_cities_Fortune_500 	for	 project_archiver
Copying Taylor Murphy response locations 	for	 taylor.murphy
Copying set2_Voronoi-diagram 	for	 project_archiver
Copying set1_major_cities 	for	 project_archiver
Copying LA 	for	 taylor.murphy
Copying NH 	for	 miller.cook

We have successfully cloned all the items from source to target. We can query the contents of one of the users below to verify:

In [28]:

user1 = target.users.search()[2]
user1

Out[28]:

Brown Rogers

Bio: None
First Name: Brown
Last Name: Rogers
Username: brown.rogers
Joined: April 04, 2017

In [30]:

user1.items()

Out[30]:

[<Item title:"AR" type:Feature Service owner:brown.rogers>,
 <Item title:"AR" type:CSV owner:brown.rogers>]

We could query the folders belonging to this user and the items within as well

In [31]:

user1.folders

Out[31]:

[{'created': 1491335086184,
  'id': 'ebfdbc2bb47f4d5f92b6673a8a68a89f',
  'title': 'Rogers_webmaps',
  'username': 'brown.rogers'}]

In [32]:

user1.items(folder=user1.folders[0]['title'])

Out[32]:

[<Item title:"Brown Rogers response locations" type:Web Map owner:brown.rogers>]

Establish relationship between items¶

So far, we have successfully cloned users, groups and items from source to target. Next, we will establish identical relationships between items as they were in the source portal.

In [33]:

RELATIONSHIP_TYPES = frozenset(['Map2Service', 'WMA2Code',
                                'Map2FeatureCollection', 'MobileApp2Code', 'Service2Data',
                                'Service2Service'])

Below, we loop through each item in source portal, find to which other item it is related and the type of that relationship. If a relationship is found, we find the corresponding items in target and establish the same relationship. To make this work, we will make use of the dictionary that maps the itemIds on source and target we created during the item clone stage. Let us take a look at that dictionary below:

In [40]:

source_target_itemId_map

Out[40]:

{'0c8ffe4ab7754eedbd2c514032f7e913': 'a83e41e4768c486d8fbece1a9cd819b1',
 '0fdcdd6eed9e4e6aa83575da5c4d8ff0': 'b7b266c064634c10868cae2daf6b922e',
 '10eff50fda644f2fa9261ee925a24495': '4328e2087aa74f92913e2af005d7590e',
 '14935a49a8544d8eb0eeadaa6216bbba': '034c8ccc02de4d199171ad5ac53decbe',
 '1506548f309140d89a19886913d8395a': '1a167bb7f305456087db833a956e0c8a',
 '1cb04f79f8a74826bab704efe5206603': 'fa725ccf2c31446d99efe6792403564d',
 '1d8820ef73064498983af1af326ddb9d': '66c5fe735bf142f0b0d79e1de0372a45',
 '28984d7a69f34b51b07a087800931d6f': 'a3e4a6ea11124ff48fa6f13100649756',
 '297a1819ba9b43ffb8adcab6a7eebd53': 'ef11e8a203014cb4899c1c61369437c6',
 '464fdece247e46b1ba861a7ac92701a4': '95f05a2413e8471c9fa06ed4b9f65f6c',
 '48f17e7559ca4d65b5eab5b78cabf085': 'be9e963a42ea4ae4872bc12059db3eeb',
 '54cf94db12e64092bef153b9f2c07176': 'ba525e9a038748dd9fe595151a7c5c73',
 '56d1921b6cfb46e89bd697668cceb708': '3db062fa1aa249efaac1ed734edf8779',
 '59132371ac22462a899bcaf7b561f1cd': '93a4bf9d217349fb9324eb2fafda5621',
 '67503e5077f240738e091e4c1da13745': 'e256401189684e1ba88ac175d01ff2d6',
 '715f8038c23b4b53a1280ae1b7b4a9bc': 'd2a560bff4034238b8b2b5278e7855ff',
 '71b66b8e628b46c9850cefd91f67f604': '18861c8f7833430fa621f566da8d055a',
 '73118cf2a8334a9884f269f81ab25c45': 'a845548e6de94b179ffab40819b8a175',
 '734529a8daad4317bc6daedfe01fdc9f': '90e2b02043df4d6a818cd3c11d9060ff',
 '75c63c8de25943f9b5ed9c477e556034': 'ae666900d38d4b0c8a610bfdc1af74c5',
 '7be7710b5e4b44cf836139a115825b2e': '6918e98589a846e7877e08a2678c52bc',
 '7c4fdc5508004253960993bbab02bf61': 'd7ded032276244cb93da101eed2a6de9',
 '7e854deebcd1442eb850a3ea84732679': '8f43bca0d1c04ab1939d7840d27d497b',
 '819d0cbed11f4f81a98e7f388fb8eeab': '9819e9b97e6b45f7b472fcea3813e9a7',
 '882009025e11471caad3f48389fde221': 'c76bab88f25843d498500b13593bc5cb',
 '8a6b1ed1b7be4cdeb298020cf5108c70': '60158b3293de4b17bd9dab7a5e49823b',
 '8d025e7368974649a3eb0691656dfa57': 'fe174f158e46437a91bd5e1bcd1208f9',
 '97d15ea9057549cdbe67344fe129c758': 'a174cddd5c2e4795a8cb3c24e3923917',
 '9afed1a827914a53af34af490f64ecdf': 'ec6adb1ba69c4b62a0a7ffc5bb77b8d0',
 'a0929a38db3240cfaffbc254d00e1827': '2544da801896434cbdee71ba53595569',
 'a89398de64444d58a9ea6a0cc407c6b7': '3bd28b9286d3406b89624d5156f42bc1',
 'aaa6ad38f9c24b4294a4c6d0994fc7c7': '985ba4a4fca9425ebdc7599ed29f4673',
 'b18c614bc00d40d0a216754d1aa8b150': 'b442463bd44341ad95858c8f5c386714',
 'b8c648204c804c289930b9e121dd8fdb': 'a90f58b560ea421193e63fddccc93b42',
 'ba3d9b3e6793406696497be34607de44': 'd2a2aa9425854b24a561b282632fde56',
 'c56e60ab29f1400fbc46cc2732724ef8': '1200541fdc444a9680e5bc447f0fb23b',
 'd65f34a670654818a010771d9a896760': '8f02739994df48e795b2b8a2b1e09dc6',
 'd6818b3d16c847b5ab9986d708b992a2': '457041ce43564d7e8304d128787bcbdb',
 'd8fdaa873f724c7abdca62a42de0c97a': '38bf01476282444e9f3468324fde8743',
 'dee8ae7fede64f8e94c39d4f6aff8d6f': '5b185e3231224f37af416f2c55d6747d',
 'e0e716ac1d774cb190765dd7a2a7421f': 'e5099e2a674f49aebf4a140920abf67a',
 'e76f3fab66534499bdc394558a645357': '9be7de598e714b66ab2058d85313e69f',
 'ea36c5dbd4e142079c43b17cb49b4258': '390da582debf43ee8dd5a2a17df816b1',
 'ebd57622fb984f1c96122c4b442ea6d7': '9f9f91d7a0e64072921f5eced954168c',
 'edba588da56b4a3ab2f2df576f9c6b99': 'e0f42fc1c8c341b9925debace5e402f9',
 'eed82e9d95dc4ad6a0f3d21e530e0a8f': '2efdd82ec6c84e9390f304b5c3240081',
 'f3f87cb385ea4ae291fd6557338e8e0b': '6fe88968317a45f9830cb9177df75c13',
 'f9866c5a2be0428abe1c7097993d9b36': '960347c4c3764e96bc15ca2817752563'}

In [41]:

for key in source_target_itemId_map.keys():
    source_item = source_items_by_id[key]
    target_itemid = source_target_itemId_map[key]
    target_item = target.content.get(target_itemid)

    print(source_item.title + " # " + source_item.type)
    for relationship in RELATIONSHIP_TYPES:
        try:
            source_related_items = source_item.related_items(relationship)
            for source_related_item in source_related_items:
                print("\t\t" + source_related_item.title + " # " + 
                      source_related_item.type +"\t## " + relationship)

                #establish same relationship amongst target items
                print("\t\t" + "establishing relationship in target portal", end=" ")
                target_related_itemid = source_target_itemId_map[source_related_item.itemid]
                target_related_item = target.content.get(target_related_itemid)
                status = target_item.add_relationship(target_related_item, relationship)
                print(str(status))
        except Exception as rel_ex:
            print("\t\t Error when checking for " + relationship + " : " + str(rel_ex))
            continue

NC # Feature Service
		NC # CSV	## Service2Data
		establishing relationship in target portal True
AR # Feature Service
		AR # CSV	## Service2Data
		establishing relationship in target portal True
set2_catalina-points # KML
FL # Feature Service
		FL # CSV	## Service2Data
		establishing relationship in target portal True
KS # Feature Service
		KS # CSV	## Service2Data
		establishing relationship in target portal True
set1_GeoJson # PDF
set2_australia # GeoJson
Smith Collins response locations # Web Map
AZ # Feature Service
		AZ # CSV	## Service2Data
		establishing relationship in target portal True
NV # Feature Service
		NV # CSV	## Service2Data
		establishing relationship in target portal True
FL # CSV
set3_Streets # Map Document
set2_counties # Locator Package
ID # Feature Service
		ID # CSV	## Service2Data
		establishing relationship in target portal True
Wilson Morgan response locations # Web Map
set1_Chicago # CSV
set2_USAcities # File Geodatabase
Jones Morris response locations # Web Map
set2_Chicago # CSV
Miller Cook response locations # Web Map
ID # CSV
set1_fortune500 # File Geodatabase
set1_gov_sites_registration # Microsoft Excel
NV # CSV
set1_india # GeoJson
Johnson Stewart response locations # Web Map
set2_SD_crime # Map Document
IN # Feature Service
		IN # CSV	## Service2Data
		establishing relationship in target portal True
set1_GeoJson # Microsoft Word
LA # Feature Service
		LA # CSV	## Service2Data
		establishing relationship in target portal True
Moore Bell response locations # Web Map
set2_empty # Map Document
set1_major_cities # Locator Package
NH # CSV
AZ # CSV
AR # CSV
IN # CSV
Brown Rogers response locations # Web Map
Davis Reed response locations # Web Map
Smith Collins response locations # Web Map
NC # CSV
set1_mapping_tech # Microsoft Powerpoint
USA_cities_Fortune_500 # Map Document
Taylor Murphy response locations # Web Map
set2_Voronoi-diagram # Microsoft Word
Williams Sanchez response locations # Web Map
LA # CSV
NH # Feature Service
		NH # CSV	## Service2Data
		establishing relationship in target portal True

Conclusion¶

Thus, with this notebook, we have successfully cloned groups, users and their contents. Note, this notebook did not copy over the services that power the service based items. Such items continue to point to the same URL as the ones in source portal did. As long as those URLs remain accessible, the web maps and layer items continue to be usable.

To run this notebook as a Python script, checkout the Python scripts in the accompanying SDK GitHub repository.

Feedback on this topic?