Sync overview

Feature layers includes a Sync capability, which when enabled, allows client applications to take feature layers offline, perform edits, and sync it back to the layer. When you checkout some features and store it offline in the client, you call that a replica. The FeatureLayerCollection class under the features module allows users to create and work with replicas. The workflow of using sync involves these three operations:

• Create replica
• Synchronize replica
• Unregister replica

To learn more about this feature, refer to the Sync Overview documentation.

Checking out data from feature layers using replicas

To create a replica, we need a feature layer that is sync enabled. We can use the syncEnabled property of a FeatureLayerCollection object to verify that. If it is compatible, it'll have a syncCapabilities property that returns a dictionary with specific sync capabilities.

We'll start by establishing a GIS connection.

# connect to a GIS
from arcgis.gis import GIS
import arcgis.features
gis = GIS() # connect to www.arcgis.com anonymously. 
            # we will use a public sync enabled feature layer

To create and work with replicas, we need to create a FeatureLayerCollection object. A FeatureLayerCollection object can be created from either a feature layer Item or directly using a feature service URL.

Here, we will connect to a sample service by Esri with URL https://sampleserver6.arcgisonline.com/arcgis/rest/services/Sync/WildfireSync/FeatureServer/.

url = 'https://sampleserver6.arcgisonline.com/arcgis/rest/services/Sync/WildfireSync/FeatureServer/'
wildfire_flc = arcgis.features.FeatureLayerCollection(url, gis)

type(wildfire_flc)

arcgis.features.layer.FeatureLayerCollection

Verify if sync is enabled

Perfect, now we have our FeatureLayerCollection object. Accessing the layers property on a FeatureLayerCollection returns a list of FeatureLayer objects. We can create a replica of one of these or all of these layers.

wildfire_flc.layers

[<FeatureLayer url:"https://sampleserver6.arcgisonline.com/arcgis/rest/services/Sync/WildfireSync/FeatureServer//0">,
 <FeatureLayer url:"https://sampleserver6.arcgisonline.com/arcgis/rest/services/Sync/WildfireSync/FeatureServer//1">,
 <FeatureLayer url:"https://sampleserver6.arcgisonline.com/arcgis/rest/services/Sync/WildfireSync/FeatureServer//2">]

# query syncEnabled property to verify is sync is enabled
wildfire_flc.properties.syncEnabled

True

# query the syncCapabilities property to view fine grained capabilities
wildfire_flc.properties.syncCapabilities

{
  "supportsRegisteringExistingData": true,
  "supportsSyncDirectionControl": true,
  "supportsPerLayerSync": true,
  "supportsPerReplicaSync": false,
  "supportsRollbackOnFailure": false,
  "supportsAsync": true,
  "supportsAttachmentsSyncDirection": true,
  "supportsSyncModelNone": true
}

List existing replicas

The replicas property on a FeatureLayerCollection object returns you a SyncManager object. This manager object contains the necessary functions for all your sync workflows. You can read more about it here.

You can determine if any replicas were created earlier on this layer by calling the get_list() method on the SyncManager object.

replica_list = wildfire_flc.replicas.get_list()
len(replica_list)

2515

As you can see, there are plenty of replicas on this layer. Let's view one:

replica_list[0]

{'replicaName': 'Ags_Fs_1543141555952',
 'replicaID': 'A2A63918-F35D-4441-8AE8-693A672F4BED'}

Create a replica

Now, we'll create our own replica of this feature layer. The create() method accepts a number of parameters allowing you to adjust what needs to be replicated and customize other options. For more information on this operation, scroll down on the SyncManager documentation to the function.

The full capability of the sync operation allows you to check out data from a feature layer, make edits, and sync the deltas (changes) back to the server in order to update the features. This is popular in use cases which involve client applications such as ArcGIS Runtime or ArcGIS Desktop applications that check out data, go offline (such as in areas where network connectivity is limited), make edits, then synchronize the data back to the server and update the features. The capability allows multiple clients to do this in parallel, thus enabling a large data collection effort.

However, if your purpose of a replica is only to check out the data (one directional), then you can verify if the extract capability is enabled on the feature layer and create a replica that is just meant for data check out. We will see this use case below:

# list all capabilities
wildfire_flc.properties.capabilities

'Create,Delete,Query,Sync,Update,Uploads,Editing'

We don't see 'Extract', which is what we need. So, let us search for a different layer via a different GIS connection (e.g. use either an existing profile or GIS(url="your enterprise", username='user name', password='password') to set up the connection)

portal_gis = GIS(url='https://pythonapi.playground.esri.com/portal', username='arcgis_python', password='amazing_arcgis_123')
search_result = portal_gis.content.search("Ports in the Western US", "Feature Layer")

search_result[0]

Ports in the Western US
Subset of ports data based on US Department of Transportation.Feature Layer Collection by api_data_owner
Last Modified: January 03, 2019
0 comments, 61 views

We'll once again create a FeatureLayerCollection object from this item

ports_flc = arcgis.features.FeatureLayerCollection.fromitem(search_result[0])
type(ports_flc)

arcgis.features.layer.FeatureLayerCollection

Verify Extract capability

ports_flc.properties.capabilities

'Query,Sync,Extract'

Now that we have a suitable feature layer (since Sync and Extract are both enabled as seen in the capabilities), we can extract the data into a file geodatabase and store it in our local file system. This happens during the replica creation. Let's dive into some more specific details of our replica before creation.

The "fullExtent" of the FeatureLayerCollection object can be extracted and used to construct the geometry filter we'll use when we call the create() method later. When the geometry_filter parameter is not specified, the create() method will go on to use the source service's full extents to contruct the geometry_filter as default.

extents = ports_flc.properties['fullExtent']
extents_str = ",".join(format(x, "10.3f") for x in [extents['xmin'],extents['ymin'],extents['xmax'], extents['ymax']])
extents_str

'-17764735.369,2241467.173,-13044640.570,5721784.017'

Now we are ready to create the geometry filter. geom_filter is a spatial filter class object from arcgis.geometry.filters module used to filter results based on a spatial relationship with geometry (in this case, using the full extents of the area of interest, or AOI).

geom_filter = {'geometryType':'esriGeometryEnvelope'}
geom_filter.update({'geometry':extents_str})
geom_filter

{'geometryType': 'esriGeometryEnvelope',
 'geometry': '-17764735.369,2241467.173,-13044640.570,5721784.017'}

Use the out_path to specify the folder path to save the replica file, e.g. executing the cell below would save the replica file at server side location /arcgis/home/data2403D6ECC5674828945916419D999BCA.geodatabase

replica1 = ports_flc.replicas.create( replica_name = 'arcgis_python_api_ports',
                                      layers='0',
                                      geometry_filter=geom_filter,
                                      sync_model="perLayer",
                                      target_type="server",
                                      data_format="sqlite",
                                      out_path=r'/arcgis/home')
replica1

{'transportType': 'esriTransportTypeURL',
 'replicaName': 'arcgis_python_api_ports',
 'responseType': 'esriReplicaResponseTypeData',
 'replicaID': '6E679BFE-90AC-453F-AFEC-06610E1EA90D',
 'targetType': 'server',
 'resultUrl': 'https://pythonapi.playground.esri.com/server/rest/directories/arcgisjobs/system/synctools_gpserver/j8f3cf4b3ee4543f39ac4e9dc22c9db1a/scratch/_ags_data2403D6ECC5674828945916419D999BCA.geodatabase',
 'submissionTime': 1554503878000,
 'lastUpdatedTime': 1554503880000,
 'status': 'Completed'}

There we go! We were able to checkout data from this feature layer into a geodatabase. Clients can use this data in any way they wish, for instance, publish it as another feature layer to a different portal or just store it for archival.

Removing replicas

The sync operation is expensive on the resources of your web GIS. Therefore, it is a good maintenance practice to remove unnecessary replicas. An ArcGIS admin could leverage the ArcGIS Python API to script and automate the process of scanning all feature layers and removing stale replicas on each of them.

A replica can be removed by calling the SyncManager's unregister() method and passing the id of a replica that needs to be removed.

# Let us query all the replicas registered on the ports feature layer from before
replica_list = ports_flc.replicas.get_list()

We know from earlier we had a ton of replicas (2515), but what if we only want to unregister the one we just made? Or ones within a certain time window? To dictate that, we can start by looping through each of the replicas returned and use the get() method to get detailed information about these replicas and look a creationDate property.

Before looping, let us take a deeper look at one of these replicas by calling the get() method:

replica1 = ports_flc.replicas.get(replica_list[0]['replicaID'])
replica1

{'replicaName': 'arcgis_python_api_ports',
 'replicaID': '6E679BFE-90AC-453F-AFEC-06610E1EA90D',
 'replicaOwner': 'arcgis_python',
 'spatialReference': {'wkid': 102100, 'latestWkid': 3857},
 'layerServerGens': [{'id': 0, 'serverGen': 1554503878473, 'serverSibGen': 0}],
 'creationDate': 1554503878473,
 'lastSyncDate': 1554503878473,
 'syncModel': 'perLayer',
 'targetType': 'server',
 'syncDirection': 'download',
 'returnsAttachments': False,
 'returnAttachments': False,
 'attachmentsSyncDirection': 'none',
 'spatialRel': 'esriSpatialRelIntersects',
 'geometry': {'xmin': -17764735.369,
  'ymin': 2241467.173,
  'xmax': -13044640.57,
  'ymax': 5721784.017,
  'spatialReference': {'wkid': 102100, 'latestWkid': 3857}},
 'layers': [{'id': 0,
   'queryOption': 'useFilter',
   'useGeometry': True,
   'includeRelated': True,
   'where': ''}]}

The creationDate key is retured as unix epoch time. We need to convert it to local time for processing:

import time
time.localtime(replica1['creationDate']/1000) #dividing by 1000 to convert micro seconds to seconds

time.struct_time(tm_year=2019, tm_mon=4, tm_mday=5, tm_hour=15, tm_min=37, tm_sec=58, tm_wday=4, tm_yday=95, tm_isdst=1)

To determine those replicas that were created 10 mins earlier, let us create an epoch timestamp for 10 mins before now and find those replicas whose time stamps are lower than this

ten_min_earlier_epoch = time.time() - 10
ten_min_earlier_epoch

1554504014.728465

Now that we have this value, we can use this and compare all of our replicas' time values. Anything less than this will be older than 10 minutes old; we can use that comparison to determine which ones to delete when we loop through.

import time
removal_list = []
for r in replica_list:
    temp_r = ports_flc.replicas.get(r['replicaID'])
    temp_dict = {'replica_id': r['replicaID'],
                'creationDate':temp_r['creationDate']/1000}
    
    #check
    if temp_dict['creationDate'] < ten_min_earlier_epoch:
        removal_list.append(temp_dict)
        print(temp_dict)

{'replica_id': '6E679BFE-90AC-453F-AFEC-06610E1EA90D', 'creationDate': 1554503878.473}

Let us loop through each of these replicas and remove them using the unregister() method:

for r in removal_list:
    result = ports_flc.replicas.unregister(r['replica_id'])
    print(result)

{'success': True}

Voila. We've mastered replicas.