Sync overview

Feature layers includes a Sync capability, which when enabled, allows client applications to take feature layers offline, perform edits and sync it back to the layer. When you checkout some features and store it offline in the client, you call that a replica. The FeatureLayerCollection class under the features module allows users to create and work with replicas. The workflow of using sync involves these three operations:

• Create replica
• Synchronize replica
• Unregister replica

To learn more about this feature, refer to the documentation here

Checking out data from feature layers using replicas

To create a replica, we need a feature layer that is sync enabled. We can use the syncEnabled property of a FeatureLayer object to verify that. Further the syncCapabilities property returns a dictionary with fine grained sync capabilities

# connect to a GIS
from arcgis.gis import GIS
import arcgis.features
gis = GIS() # connect to www.arcgis.com anonymously. 
            # we will use a public sync enabled feature layer

To create and work with replicas, we need to create a FeatureLayerCollection object. A FeatureLayerCollection object can be created from either a feature layer Item or directly using a feature service URL.

Here, we will connect to a sample service by esri with URL https://sampleserver6.arcgisonline.com/arcgis/rest/services/Sync/WildfireSync/FeatureServer/

url = 'https://sampleserver6.arcgisonline.com/arcgis/rest/services/Sync/WildfireSync/FeatureServer/'
wildfire_flc = arcgis.features.FeatureLayerCollection(url, gis)

type(wildfire_flc)

arcgis.features.layer.FeatureLayerCollection

Verify if sync is enabled

Accessing the layers property on a FeatureLayerCollection returns a list of FeatureLayer objects. We can create a replica of one of these or all of these layers

wildfire_flc.layers

[<FeatureLayer url:"https://sampleserver6.arcgisonline.com/arcgis/rest/services/Sync/WildfireSync/FeatureServer//0">,
 <FeatureLayer url:"https://sampleserver6.arcgisonline.com/arcgis/rest/services/Sync/WildfireSync/FeatureServer//1">,
 <FeatureLayer url:"https://sampleserver6.arcgisonline.com/arcgis/rest/services/Sync/WildfireSync/FeatureServer//2">]

# query syncEnabled property to verify is sync is enabled
wildfire_flc.properties.syncEnabled

True

# query the syncCapabilities property to view fine grained capabilities
wildfire_flc.properties.syncCapabilities

{
  "supportsRegisteringExistingData": true,
  "supportsSyncDirectionControl": true,
  "supportsPerLayerSync": true,
  "supportsPerReplicaSync": false,
  "supportsRollbackOnFailure": false,
  "supportsAsync": true,
  "supportsAttachmentsSyncDirection": true,
  "supportsSyncModelNone": true
}

List existing replicas

The replicas property on a FeatureLayerCollection object returns you a SyncManager object. You would work with this manager object for all your sync workflows.

You can find if any replicas were created earlier on this layer by calling get_list() method on the SyncManager object.

replica_list = wildfire_flc.replicas.get_list()
len(replica_list)

2515

As you can see, there are plenty of replicas on this layer. Let us view one of it

replica_list[0]

{'replicaName': 'Ags_Fs_1543141555952',
 'replicaID': 'A2A63918-F35D-4441-8AE8-693A672F4BED'}

Create a replica

Now, let us create our own replica of this feature layer. The create() method accepts a number of parameters allowing you to adjust what needs to be replicated and customize other options. For more information on this operation, refer to the documentation here.

The full capability of the sync operation allows you to check out data from a feature layer, make edits and sync the deltas (changes) back to the server and update the features. This is popular in use cases which involve client applications such as ArcGIS Runtime or ArcGIS Desktop applications check out data, go offline (such as in areas where network connectivity is limited), make edits, then synchronize the data back to the server and update the features. The capability allows multiple clients to do this in parallel, thus enabling a large data collection effort.

However, if your purpose of a replica is only to check out the data (one directional), then you can verify if the extract capability is enabled on the feature layer and create a replica that is just meant for data check out. We will see this use case below:

# list all capabilities
wildfire_flc.properties.capabilities

'Create,Delete,Query,Sync,Update,Uploads,Editing'

This layer has disabled 'Extract'. Hence let us search for a different layer via a different GIS connection (e.g. use either an existing profile or GIS(url="your enterprise", username='user name', password='password') to set up the connection)

portal_gis = GIS(url='https://pythonapi.playground.esri.com/portal', username='arcgis_python', password='amazing_arcgis_123')
search_result = portal_gis.content.search("Ports in the Western US", "Feature Layer")

search_result[0]

Ports in the Western US
Subset of ports data based on US Department of Transportation.Feature Layer Collection by api_data_owner
Last Modified: January 03, 2019
0 comments, 61 views

Let us create a FeatureLayerCollection object from this item

ports_flc = arcgis.features.FeatureLayerCollection.fromitem(search_result[0])
type(ports_flc)

arcgis.features.layer.FeatureLayerCollection

Verify Extract capability

ports_flc.properties.capabilities

'Query,Sync,Extract'

This is a suitable feature layer (since Sync and Extract are both enabled as seen in the capabilities), let us extract the data into a file geodatabase and store it in local file system

ports_flc = arcgis.features.FeatureLayerCollection.fromitem(search_result[0])

The "fullExtent" of the speicifc FeatureLayerCollection object can be extracted and used to construct the geometry filter to be used in the create method mentioned later. When the geometry_filter parameter is not specified, the create method will go on to use the source service's full extents to contruct the geometry_filter as default.

extents = ports_flc.properties['fullExtent']
extents_str = ",".join(format(x, "10.3f") for x in [extents['xmin'],extents['ymin'],extents['xmax'], extents['ymax']])
extents_str

'-17764735.369,2241467.173,-13044640.570,5721784.017'

Now we are ready to create the geometry filter. geom_filter is a spatial filter class object from arcgis.geometry.filters module to filter results by a spatial relationship with geometry (in this case, using the full extents of the AOI)

geom_filter = {'geometryType':'esriGeometryEnvelope'}
geom_filter.update({'geometry':extents_str})
geom_filter

{'geometryType': 'esriGeometryEnvelope',
 'geometry': '-17764735.369,2241467.173,-13044640.570,5721784.017'}

Use the out_path to specify the folder path to save the replica file, e.g. executing the cell below would save the replica file at server side location /arcgis/home/data2403D6ECC5674828945916419D999BCA.geodatabase

replica1 = ports_flc.replicas.create( replica_name = 'arcgis_python_api_ports',
                                      layers='0',
                                      geometry_filter=geom_filter,
                                      sync_model="perLayer",
                                      target_type="server",
                                      data_format="sqlite",
                                      out_path=r'/arcgis/home')
replica1

{'transportType': 'esriTransportTypeURL',
 'replicaName': 'arcgis_python_api_ports',
 'responseType': 'esriReplicaResponseTypeData',
 'replicaID': '6E679BFE-90AC-453F-AFEC-06610E1EA90D',
 'targetType': 'server',
 'resultUrl': 'https://pythonapi.playground.esri.com/server/rest/directories/arcgisjobs/system/synctools_gpserver/j8f3cf4b3ee4543f39ac4e9dc22c9db1a/scratch/_ags_data2403D6ECC5674828945916419D999BCA.geodatabase',
 'submissionTime': 1554503878000,
 'lastUpdatedTime': 1554503880000,
 'status': 'Completed'}

Thus, we were able to checkout data from this feature layer into a geodatabase. Clients can use this data in any way they wish, for instance, publish it as another feature layer to a different portal or just store it for archival.

Removing replicas

The sync operation is expensive on the resources of your web GIS. Hence, it is a good maintenance practice to remove unnecessary replicas. An ArcGIS admin could use the ArcGIS Python API to script and automate the process of scanning all feature layers and removing stale replicas on each of them.

A replica can be removed by calling the unregister() method and passing the id of a replica that needs to be removed.

# Let us query all the replicas registered on the ports feature layer from before
replica_list = ports_flc.replicas.get_list()

for r in replica_list:
    print(r)

{'replicaName': 'arcgis_python_api_ports', 'replicaID': '6E679BFE-90AC-453F-AFEC-06610E1EA90D'}

There are more than a few. I only want to remove the replicas that were registered 10 minutes ago. But, your search criteria could be any other.

We will loop through each of the replicas returned and use the get() method to get detailed information about these replicas and look a creationDate property.

Before looping, let us take a deeper look at one of these replicas by calling the get() method:

replica1 = ports_flc.replicas.get(replica_list[0]['replicaID'])
replica1

{'replicaName': 'arcgis_python_api_ports',
 'replicaID': '6E679BFE-90AC-453F-AFEC-06610E1EA90D',
 'replicaOwner': 'arcgis_python',
 'spatialReference': {'wkid': 102100, 'latestWkid': 3857},
 'layerServerGens': [{'id': 0, 'serverGen': 1554503878473, 'serverSibGen': 0}],
 'creationDate': 1554503878473,
 'lastSyncDate': 1554503878473,
 'syncModel': 'perLayer',
 'targetType': 'server',
 'syncDirection': 'download',
 'returnsAttachments': False,
 'returnAttachments': False,
 'attachmentsSyncDirection': 'none',
 'spatialRel': 'esriSpatialRelIntersects',
 'geometry': {'xmin': -17764735.369,
  'ymin': 2241467.173,
  'xmax': -13044640.57,
  'ymax': 5721784.017,
  'spatialReference': {'wkid': 102100, 'latestWkid': 3857}},
 'layers': [{'id': 0,
   'queryOption': 'useFilter',
   'useGeometry': True,
   'includeRelated': True,
   'where': ''}]}

The creationDate key is retured as unix epoch time. We need to convert it to local time for processing:

import time
time.localtime(replica1['creationDate']/1000) #dividing by 1000 to convert micro seconds to seconds

time.struct_time(tm_year=2019, tm_mon=4, tm_mday=5, tm_hour=15, tm_min=37, tm_sec=58, tm_wday=4, tm_yday=95, tm_isdst=1)

To determine those replicas that were created 10 mins earlier, let us create an epoch timestamp for 10 mins before now and find those replicas whose time stamps are lower than this

ten_min_earlier_epoch = time.time() - 10
ten_min_earlier_epoch

1554504014.728465

import time
removal_list = []
for r in replica_list:
    temp_r = ports_flc.replicas.get(r['replicaID'])
    temp_dict = {'replica_id': r['replicaID'],
                'creationDate':temp_r['creationDate']/1000}
    
    #check
    if temp_dict['creationDate'] < ten_min_earlier_epoch:
        removal_list.append(temp_dict)
        print(temp_dict)

{'replica_id': '6E679BFE-90AC-453F-AFEC-06610E1EA90D', 'creationDate': 1554503878.473}

Let us loop through each of these replicas and remove them using the unregister() method:

for r in removal_list:
    result = ports_flc.replicas.unregister(r['replica_id'])
    print(result)

{'success': True}