Updating features in a feature layer

As content publishers, you may be required to keep certain web layers up to date. As new data arrives, you may have to append new features, update existing features etc. There are a couple of different options to accomplish this:

  • Method 1: editing individual features as updated datasets are available
  • Method 2: overwriting feature layers altogether with updated datasets

Depending on the number of features that are updated, your workflow requirements, you may adopt either or both kinds of update mechanisms.

In this sample, we explore the first method:

Method 1

For Method 2, refer to the sample titled Overwriting feature layers

Note: To run this sample, you need the pandas library in your conda environment. If you don't have the library, install it by running the following command from cmd.exe or your shell

conda install pandas
In [ ]:
# Connect to the GIS
from arcgis.gis import GIS
from arcgis import features
from getpass import getpass #to accept passwords in an interactive fashion
import pandas as pd
In [ ]:
#Access the portal using "amazing_arcgis_123" as password for the given Username. 
password = getpass()
gis = GIS("https://python.playground.esri.com/portal", "arcgis_python", password)
········

Updating feature layer by editing individual features

Let us consider a scenario where we need to update a feature layer containing the capital cities of the US. We have 3 csv datasets simulating an update workflow as described below:

  1. capitals_1.csv -- contains the initial, incomplete dataset
  2. capitals_2.csv -- contains additional points and updates to existing points, building on top of capitals_1.csv
  3. capitals_annex.csv -- an alternate table containing additional attribute information

Our goal is to update the feature layer with each of these datasets doing the necessary edit operations.

Publish the cities feature layer using the initial dataset

In [ ]:
# read the initial csv
csv1 = 'data/updating_gis_content/capitals_1.csv'
cities_df_1 = pd.read_csv(csv1)
cities_df_1.head()
Out[ ]:
city_id name state capital pop2000 pop2007 longitude latitude
0 1 Honolulu HI State 371657 378587 -157.823436 21.305782
1 2 Juneau AK State 30711 31592 -134.511582 58.351418
2 3 Boise City ID State 185787 203529 -116.237655 43.613736
3 4 Olympia WA State 27514 45523 -122.893073 47.042418
4 5 Salem OR State 136924 152039 -123.029155 44.931109
In [ ]:
# print the number of records in this csv
cities_df_1.shape
Out[ ]:
(19, 8)

As you can see, this dataset only contains 19 rows or 19 capital cities. It is not the complete dataset.

In [ ]:
# add the initial csv file and publish that as a web layer
item_prop = {'title':'USA Capitals spreadsheet'}
csv_item = gis.content.add(item_properties=item_prop, data=csv1)
csv_item
Out[ ]:
USA Capitals spreadsheet
CSV by arcgis_python
Last Modified: June 23, 2017
0 comments, 0 views

This spreadsheet has co-ordinates as latitude and longitude columns which will be used for geometries during publishing.

In [ ]:
# publish the csv item into a feature layer
cities_item = csv_item.publish()
cities_item
Out[ ]:
USA Capitals spreadsheet
Feature Layer Collection by arcgis_python
Last Modified: June 23, 2017
0 comments, 0 views
In [ ]:
# update the item metadata
item_prop = {'title':'USA Capitals'}
cities_item.update(item_properties = item_prop, thumbnail='data/updating_gis_content/capital_cities.png')
cities_item
Out[ ]:
USA Capitals
Feature Layer Collection by arcgis_python
Last Modified: June 23, 2017
0 comments, 0 views

Apply updates from the second spreadsheet

The next set of updates have arrived and are stored in capitals_2.csv. We are told it contains corrections for the original set in addition to new features. We need to figure out which rows have changed, apply update operation on those, then apply add operation to new rows.

To start with, let us read the second csv file. Note, in this sample, data is stored in csv. In reality, it could be from your enterprise database or any other data source.

In [ ]:
# read the second csv set
csv2 = 'data/updating_gis_content/capitals_2.csv'
cities_df_2 = pd.read_csv(csv2)
cities_df_2.head()
Out[ ]:
city_id name state capital pop2000 pop2007 longitude latitude
0 20 Baton Rouge LA State 227818 228810 -91.140227 30.458091
1 21 Helena MT State 25780 26007 -112.027027 46.595809
2 22 Bismarck ND State 55532 59344 -100.779000 46.813346
3 23 Pierre SD State 13876 14169 -100.336382 44.367964
4 24 St. Paul MN State 287151 291643 -93.114118 44.954364
In [ ]:
# get the dimensions of this csv
cities_df_2.shape
Out[ ]:
(36, 8)

Identifying existing features that need to be updated

To identify features that need to be updated, let us read the attribute table of the published feature layer and compare that against the second csv. To read the attribute table, we perform a query() on the feature layer which returns us an arcgis.feature.FeatureSet object. Refer to the guide pages on accessing features from feature layers to learn more about this.

Note, at this point, we could work with the cities_df_1 dataframe we created from the original csv file. However, in practice you may not always have the original dataset or your feature layer might have undergone edits after it was published. Hence, we query the feature layer directly.

In [ ]:
cities_flayer = cities_item.layers[0]
cities_fset = cities_flayer.query() #querying without any conditions returns all the features
cities_fset.df.head()
Out[ ]:
capital city_id latitude longitude name objectid pop2000 pop2007 state SHAPE
0 State 1 21.305782 -157.823436 Honolulu 1 371657 378587 HI {'x': -17568824.553, 'y': 2428377.352700006}
1 State 2 58.351418 -134.511582 Juneau 2 30711 31592 AK {'x': -14973760.769500002, 'y': 8041504.674200...
2 State 3 43.613736 -116.237655 Boise City 3 185787 203529 ID {'x': -12939516.521100001, 'y': 5405860.248099...
3 State 4 47.042418 -122.893073 Olympia 4 27514 45523 WA {'x': -13680394.263900002, 'y': 5949000.547900...
4 State 5 44.931109 -123.029155 Salem 5 136924 152039 OR {'x': -13695542.842799995, 'y': 5610682.544100...

The city_id column is common between both the datasets. Next, let us perform an inner join with the table from feature layer as left and updated csv as right. Inner joins will yield those rows that are present in both tables. Learn more about inner joins here.

In [ ]:
overlap_rows = pd.merge(left = cities_fset.df, right = cities_df_2, how='inner',
                       on = 'city_id')
overlap_rows
Out[ ]:
capital_x city_id latitude_x longitude_x name_x objectid pop2000_x pop2007_x state_x SHAPE name_y state_y capital_y pop2000_y pop2007_y longitude_y latitude_y
0 State 4 47.042418 -122.893073 Olympia 4 27514 45523 WA {'x': -13680394.263900002, 'y': 5949000.547900... Olympia WA State 42514 45523 -122.893073 47.042418
1 State 10 -41.145545 104.802046 Cheyenne 10 53011 54750 WY {'x': 11666510.350300007, 'y': -5033833.302499... Cheyenne WY State 53011 54750 -104.802046 41.145545
2 State 13 35.482309 -97.534991 Oklahoma City 13 506132 552556 OKK {'x': -10857545.543799996, 'y': 4229619.674200... Oklahoma City OK State 506132 552556 -97.534991 35.482309
3 State 16 41.590936 -93.620864 Des Moines 16 200682 201257 IA {'x': -10421826.864700003, 'y': 5099899.263700... Des Moines IA State 198682 201257 -93.620864 41.590936

Thus, of 19 features in original and 36 features in second csv, 4 features are common. Inspecting the table, we find certain columns are updated, for instance, Cheyenne has its coordinates corrected, Oklahoma City has its state abbreviation corrected and similarly other cities have one of their attribute columns updated.

We could either update individual attribute values for these 4 features or update all attribute values with the latest csv. Below, we are performing the latter as it is simple and fast.

Perform updates to the existing features

In [ ]:
features_for_update = [] #list containing corrected features
all_features = cities_fset.features
In [ ]:
# inspect one of the features
all_features[0]
Out[ ]:
{"attributes": {"latitude": 21.30578163, "pop2000": 371657, "pop2007": 378587, "city_id": 1, "longitude": -157.8234362, "objectid": 1, "state": "HI", "name": "Honolulu", "capital": "State"}, "geometry": {"y": 2428377.352700006, "x": -17568824.553}}

Note the X and Y geometry values are different from decimal degree coordinates present in Longitude and Latitude fields. To perform geometry edits, we need to project the coordinates to match that of the feature layer.

In [ ]:
# get the spatial reference of the features since we need to update the geometry
cities_fset.spatial_reference
Out[ ]:
{'latestWkid': 3857, 'wkid': 102100}

Below, we prepare updated geometries and attributes for each of the 4 features we determined above. We use the arcgis.geometry module to project the coordinates from geographic to projected coordinate system. The cell below prints the original Feature objects followed by the updated ones. If you look closely, you can find the differences.

In [ ]:
from arcgis import geometry #use geometry module to project Long,Lat to X and Y
from copy import deepcopy

for city_id in overlap_rows['city_id']:
    # get the feature to be updated
    original_feature = [f for f in all_features if f.attributes['city_id'] == city_id][0]
    feature_to_be_updated = deepcopy(original_feature)
    
    print(str(original_feature))
    
    # get the matching row from csv
    matching_row = cities_df_2.where(cities_df_2.city_id == city_id).dropna()
    
    #get geometries in the destination coordinate system
    input_geometry = {'y':float(matching_row['latitude']),
                       'x':float(matching_row['longitude'])}
    output_geometry = geometry.project(geometries = [input_geometry],
                                       in_sr = 4326, 
                                       out_sr = cities_fset.spatial_reference['latestWkid'],
                                      gis = gis)
    
    # assign the updated values
    feature_to_be_updated.geometry = output_geometry[0]
    feature_to_be_updated.attributes['longitude'] = float(matching_row['longitude'])
    feature_to_be_updated.attributes['city_id'] = int(matching_row['city_id'])
    feature_to_be_updated.attributes['state'] = matching_row['state'].values[0]
    feature_to_be_updated.attributes['capital'] = matching_row['capital'].values[0]
    feature_to_be_updated.attributes['latitude'] = float(matching_row['latitude'])
    feature_to_be_updated.attributes['name'] = matching_row['name'].values[0]
    feature_to_be_updated.attributes['pop2000'] = int(matching_row['pop2000'])
    feature_to_be_updated.attributes['pop2007'] = int(matching_row['pop2007'])
    
    #add this to the list of features to be updated
    features_for_update.append(feature_to_be_updated)
    
    print(str(feature_to_be_updated))
    print("========================================================================")
{"attributes": {"latitude": 47.04241817, "pop2000": 27514, "pop2007": 45523, "city_id": 4, "longitude": -122.8930726, "objectid": 4, "state": "WA", "name": "Olympia", "capital": "State"}, "geometry": {"y": 5949000.547900006, "x": -13680394.263900002}}
{"attributes": {"latitude": 47.04241817, "pop2000": 42514, "pop2007": 45523, "city_id": 4, "longitude": -122.8930726, "capital": "State", "state": "WA", "name": "Olympia", "objectid": 4}, "geometry": {"x": -13680394.2638528, "y": 5949000.54792491}}
========================================================================
{"attributes": {"latitude": -41.14554516, "pop2000": 53011, "pop2007": 54750, "city_id": 10, "longitude": 104.8020456, "objectid": 10, "state": "WY", "name": "Cheyenne", "capital": "State"}, "geometry": {"y": -5033833.302499998, "x": 11666510.350300007}}
{"attributes": {"latitude": 41.14554516, "pop2000": 53011, "pop2007": 54750, "city_id": 10, "longitude": -104.80204559999999, "capital": "State", "state": "WY", "name": "Cheyenne", "objectid": 10}, "geometry": {"x": -11666510.350285435, "y": 5033833.302497153}}
========================================================================
{"attributes": {"latitude": 35.48230867, "pop2000": 506132, "pop2007": 552556, "city_id": 13, "longitude": -97.5349911, "objectid": 13, "state": "OKK", "name": "Oklahoma City", "capital": "State"}, "geometry": {"y": 4229619.674200006, "x": -10857545.543799996}}
{"attributes": {"latitude": 35.48230867, "pop2000": 506132, "pop2007": 552556, "city_id": 13, "longitude": -97.5349911, "capital": "State", "state": "OK", "name": "Oklahoma City", "objectid": 13}, "geometry": {"x": -10857545.54377847, "y": 4229619.674165817}}
========================================================================
{"attributes": {"latitude": 41.59093617, "pop2000": 200682, "pop2007": 201257, "city_id": 16, "longitude": -93.62086361, "objectid": 16, "state": "IA", "name": "Des Moines", "capital": "State"}, "geometry": {"y": 5099899.263700008, "x": -10421826.864700003}}
{"attributes": {"latitude": 41.59093617, "pop2000": 198682, "pop2007": 201257, "city_id": 16, "longitude": -93.62086361, "capital": "State", "state": "IA", "name": "Des Moines", "objectid": 16}, "geometry": {"x": -10421826.864691716, "y": 5099899.263692743}}
========================================================================

We have constructed a list of features with updated values. We can use this list to perform updates on the feature layer.

In [ ]:
features_for_update
Out[ ]:
[{"attributes": {"latitude": 47.04241817, "pop2000": 42514, "pop2007": 45523, "city_id": 4, "longitude": -122.8930726, "capital": "State", "state": "WA", "name": "Olympia", "objectid": 4}, "geometry": {"x": -13680394.2638528, "y": 5949000.54792491}},
 {"attributes": {"latitude": 41.14554516, "pop2000": 53011, "pop2007": 54750, "city_id": 10, "longitude": -104.80204559999999, "capital": "State", "state": "WY", "name": "Cheyenne", "objectid": 10}, "geometry": {"x": -11666510.350285435, "y": 5033833.302497153}},
 {"attributes": {"latitude": 35.48230867, "pop2000": 506132, "pop2007": 552556, "city_id": 13, "longitude": -97.5349911, "capital": "State", "state": "OK", "name": "Oklahoma City", "objectid": 13}, "geometry": {"x": -10857545.54377847, "y": 4229619.674165817}},
 {"attributes": {"latitude": 41.59093617, "pop2000": 198682, "pop2007": 201257, "city_id": 16, "longitude": -93.62086361, "capital": "State", "state": "IA", "name": "Des Moines", "objectid": 16}, "geometry": {"x": -10421826.864691716, "y": 5099899.263692743}}]

To update the feature layer, call the edit_features() method of the FeatureLayer object and pass the list of features to the updates parameter:

In [ ]:
cities_flayer.edit_features(updates= features_for_update)
Out[ ]:
{'addResults': [],
 'attachments': {'addResults': [], 'deleteResults': [], 'updateResults': []},
 'deleteResults': [],
 'updateResults': [{'globalId': None, 'objectId': 4, 'success': True},
  {'globalId': None, 'objectId': 10, 'success': True},
  {'globalId': None, 'objectId': 13, 'success': True},
  {'globalId': None, 'objectId': 16, 'success': True}]}

We have successfully applied corrections to those features which existed in the feature layer from the initial dataset. Next let us proceed to adding new features present only in the second csv file.

Identifying new features that need to be added

In [ ]:
#select those rows in the capitals_2.csv that do not overlap with those in capitals_1.csv
new_rows = cities_df_2[~cities_df_2['city_id'].isin(overlap_rows['city_id'])]
print(new_rows.shape)
(32, 8)
In [ ]:
new_rows.head()
Out[ ]:
city_id name state capital pop2000 pop2007 longitude latitude
0 20 Baton Rouge LA State 227818 228810 -91.140227 30.458091
1 21 Helena MT State 25780 26007 -112.027027 46.595809
2 22 Bismarck ND State 55532 59344 -100.779000 46.813346
3 23 Pierre SD State 13876 14169 -100.336382 44.367964
4 24 St. Paul MN State 287151 291643 -93.114118 44.954364

Thus, of the total 36 rows in the second csv, we have determined the 32 other rows which are new and need to be appended as new features.

Adding new features

Next, let us compose another list of Feature objects similar to earlier, from the new_rows data frame.

In [ ]:
features_to_be_added = []

# get a template feature object
template_feature = deepcopy(features_for_update[0])

# loop through each row and add to the list of features to be added
for row in new_rows.iterrows():
    new_feature = deepcopy(template_feature)
    
    #print
    print("Creating " + row[1]['name'])
    
    #get geometries in the destination coordinate system
    input_geometry = {'y':float(row[1]['latitude']),
                       'x':float(row[1]['longitude'])}
    output_geometry = geometry.project(geometries = [input_geometry],
                                       in_sr = 4326, 
                                       out_sr = cities_fset.spatial_reference['latestWkid'],
                                      gis = gis)
    
    # assign the updated values
    new_feature.geometry = output_geometry[0]
    new_feature.attributes['longitude'] = float(row[1]['longitude'])
    new_feature.attributes['city_id'] = int(row[1]['city_id'])
    new_feature.attributes['state'] = row[1]['state']
    new_feature.attributes['capital'] = row[1]['capital']
    new_feature.attributes['latitude'] = float(row[1]['latitude'])
    new_feature.attributes['name'] = row[1]['name']
    new_feature.attributes['pop2000'] = int(row[1]['pop2000'])
    new_feature.attributes['pop2007'] = int(row[1]['pop2007'])
    
    #add this to the list of features to be updated
    features_to_be_added.append(new_feature)
Creating Baton Rouge
Creating Helena
Creating Bismarck
Creating Pierre
Creating St. Paul
Creating Madison
Creating Lansing
Creating Augusta
Creating Montpelier
Creating Albany
Creating Boston
Creating Concord
Creating Tallahassee
Creating Jackson
Creating Nashville
Creating Montgomery
Creating Springfield
Creating Indianapolis
Creating Frankfort
Creating Columbus
Creating Charleston
Creating Atlanta
Creating Columbia
Creating Raleigh
Creating Richmond
Creating Harrisburg
Creating Trenton
Creating Dover
Creating Washington
Creating Annapolis
Creating Hartford
Creating Providence
In [ ]:
# take a look at one of the features we created
features_to_be_added[0]
Out[ ]:
{"attributes": {"latitude": 30.45809113, "pop2000": 227818, "pop2007": 228810, "city_id": 20, "longitude": -91.14022709999999, "capital": "State", "state": "LA", "name": "Baton Rouge", "objectid": 4}, "geometry": {"x": -10145683.671555312, "y": 3562570.141302621}}

Thus, we have created a list of Feature objects with appropriate attributes and geometries. Next, to add these new features to the feature layer, call the edit_features() method of the FeatureLayer object and pass the list of Feature objects to the adds parameter:

In [ ]:
cities_flayer.edit_features(adds = features_to_be_added)
Out[ ]:
{'addResults': [{'globalId': None, 'objectId': 20, 'success': True},
  {'globalId': None, 'objectId': 21, 'success': True},
  {'globalId': None, 'objectId': 22, 'success': True},
  {'globalId': None, 'objectId': 23, 'success': True},
  {'globalId': None, 'objectId': 24, 'success': True},
  {'globalId': None, 'objectId': 25, 'success': True},
  {'globalId': None, 'objectId': 26, 'success': True},
  {'globalId': None, 'objectId': 27, 'success': True},
  {'globalId': None, 'objectId': 28, 'success': True},
  {'globalId': None, 'objectId': 29, 'success': True},
  {'globalId': None, 'objectId': 30, 'success': True},
  {'globalId': None, 'objectId': 31, 'success': True},
  {'globalId': None, 'objectId': 32, 'success': True},
  {'globalId': None, 'objectId': 33, 'success': True},
  {'globalId': None, 'objectId': 34, 'success': True},
  {'globalId': None, 'objectId': 35, 'success': True},
  {'globalId': None, 'objectId': 36, 'success': True},
  {'globalId': None, 'objectId': 37, 'success': True},
  {'globalId': None, 'objectId': 38, 'success': True},
  {'globalId': None, 'objectId': 39, 'success': True},
  {'globalId': None, 'objectId': 40, 'success': True},
  {'globalId': None, 'objectId': 41, 'success': True},
  {'globalId': None, 'objectId': 42, 'success': True},
  {'globalId': None, 'objectId': 43, 'success': True},
  {'globalId': None, 'objectId': 44, 'success': True},
  {'globalId': None, 'objectId': 45, 'success': True},
  {'globalId': None, 'objectId': 46, 'success': True},
  {'globalId': None, 'objectId': 47, 'success': True},
  {'globalId': None, 'objectId': 48, 'success': True},
  {'globalId': None, 'objectId': 49, 'success': True},
  {'globalId': None, 'objectId': 50, 'success': True},
  {'globalId': None, 'objectId': 51, 'success': True}],
 'attachments': {'addResults': [], 'deleteResults': [], 'updateResults': []},
 'deleteResults': [],
 'updateResults': []}

Thus, we have successfully applied edits from second csv file. Next let us look at how we can apply edits from third csv file.

Apply edits from third spreadsheet

The next set of updates have arrived and are stored in capitals_annex.csv. We are told it contains additional columns for each of the features that we want to add to the feature layer.

To start with, let us read the third csv file. Note in this sample, data is stored in csv. In reality, it could be from your enterprise database or any other data source.

In [ ]:
# read the third csv set
csv3 = 'data/updating_gis_content/capitals_annex.csv'
cities_df_3 = pd.read_csv(csv3)
cities_df_3.head()
Out[ ]:
city_id name class white black ameri_es asian hawn_pl hispanic males females
0 1 Honolulu Census Designated Place 73093 6038 689 207588 25457 16229 182628 189029
1 2 Juneau City and Borough 22969 248 3496 1438 116 1040 15469 15242
2 3 Boise City City 171204 1437 1300 3870 302 8410 92014 93773
3 4 Olympia City 36246 805 553 2473 125 1863 20319 22195
4 5 Salem City 113746 1750 2064 3304 643 19973 68752 68172
In [ ]:
#find the number of rows in the third csv
cities_df_3.shape
Out[ ]:
(51, 11)

The capitals_annex.csv does not add new features, instead it adds additional attribute columns to existing features. It has 51 rows which were found to match the 19 + 32 rows from first and second csv files. The columns City_ID and NAME are common to all 3 spreadsheets. Next let us take a look at how we can append this additional attribute information to our feature layer.

Inspecting existing fields of the feature layer

The manager property of the FeatureLayer object exposes a set of methods to read and update the properties and definition of feature layers.

In [ ]:
#Get the existing list of fields on the cities feature layer
cities_fields = cities_flayer.manager.properties.fields

# Your feature layer may have multiple fields, 
# instead of printing all, let us take a look at one of the fields:
cities_fields[1]
Out[ ]:
{
  "alias": "city_id",
  "name": "city_id",
  "type": "esriFieldTypeInteger",
  "length": 10,
  "editable": true,
  "nullable": true,
  "domain": null,
  "sqlType": "sqlTypeInteger"
}

From above, we can see the representation of one of the fields. Let us loop through each of the fields and print the name, alias, type and sqlType properties

In [ ]:
for field in cities_fields:
    print(field.name, "\t|\t", field.alias, "\t|\t", field.type, "\t|\t", field.sqlType)
objectid 	|	 objectid 	|	 esriFieldTypeOID 	|	 sqlTypeInteger
city_id 	|	 city_id 	|	 esriFieldTypeInteger 	|	 sqlTypeInteger
name 	|	 name 	|	 esriFieldTypeString 	|	 sqlTypeVarchar
state 	|	 state 	|	 esriFieldTypeString 	|	 sqlTypeVarchar
capital 	|	 capital 	|	 esriFieldTypeString 	|	 sqlTypeVarchar
pop2000 	|	 pop2000 	|	 esriFieldTypeInteger 	|	 sqlTypeInteger
pop2007 	|	 pop2007 	|	 esriFieldTypeInteger 	|	 sqlTypeInteger
longitude 	|	 longitude 	|	 esriFieldTypeDouble 	|	 sqlTypeNumeric
latitude 	|	 latitude 	|	 esriFieldTypeDouble 	|	 sqlTypeNumeric

Preparing additional columns to add to the feature layer

Now that we have an idea of how the fields are defined, we can go ahead and append new fields to the layer's definition. Once we compose a list of new fields, by calling the add_to_definition() method we can push those changes to the feature layer. Once the feature layer's definition is updated with new fields, we can loop through each feature and add the appropriate attribute values.

To compose a list of new fields to be added, we start by making a copy of one of the fields as a template and start editing it. One easy part in this example is, all new fields that need to be added except one, are of the same data type: integer. With your data, this may not be the case. In such instances, you can add each field individually.

In [ ]:
# get a template field
template_field = dict(deepcopy(cities_fields[1]))
template_field
Out[ ]:
{'alias': 'city_id',
 'domain': None,
 'editable': True,
 'length': 10,
 'name': 'city_id',
 'nullable': True,
 'sqlType': 'sqlTypeInteger',
 'type': 'esriFieldTypeInteger'}

Let us use pandas to get the list of fields that are new in spread sheet 3

In [ ]:
# get the list of new fields to add from the third spreadsheet, that are not in spread sheets 1,2
new_field_names = list(cities_df_3.columns.difference(cities_df_1.columns))
new_field_names
Out[ ]:
['ameri_es',
 'asian',
 'black',
 'class',
 'females',
 'hawn_pl',
 'hispanic',
 'males',
 'white']

Now loop though each new field name and create a field dictionary using the template we created earlier. Except the field titled class all other fields are of type integer.

In [ ]:
fields_to_be_added = []
for new_field_name in new_field_names:
    current_field = deepcopy(template_field)
    if new_field_name.lower() == 'class':
        current_field['sqlType'] = 'sqlTypeVarchar'
        current_field['type'] = 'esriFieldTypeString'
        current_field['length'] = 8000
        
    current_field['name'] = new_field_name.lower()
    current_field['alias'] = new_field_name
    fields_to_be_added.append(current_field)
    
len(fields_to_be_added)
Out[ ]:
9
In [ ]:
#inspect one of the fields
fields_to_be_added[3]
Out[ ]:
{'alias': 'class',
 'domain': None,
 'editable': True,
 'length': 8000,
 'name': 'class',
 'nullable': True,
 'sqlType': 'sqlTypeVarchar',
 'type': 'esriFieldTypeString'}

Adding additional fields to the feature layer

The list of new fields we composed can be pushed to the server by calling add_to_definition() method on the manager property.

In [ ]:
cities_flayer.manager.add_to_definition({'fields':fields_to_be_added})
Out[ ]:
{'success': True}

Thus, we have successfully added new fields to our feature layer. Let us verify the new columns show up:

In [ ]:
new_cities_fields = cities_flayer.manager.properties.fields
len(new_cities_fields)
Out[ ]:
18
In [ ]:
for field in new_cities_fields:
    print(field.name + "\t|\t", field.type)
objectid	|	 esriFieldTypeOID
city_id	|	 esriFieldTypeInteger
name	|	 esriFieldTypeString
state	|	 esriFieldTypeString
capital	|	 esriFieldTypeString
pop2000	|	 esriFieldTypeInteger
pop2007	|	 esriFieldTypeInteger
longitude	|	 esriFieldTypeDouble
latitude	|	 esriFieldTypeDouble
ameri_es	|	 esriFieldTypeInteger
asian	|	 esriFieldTypeInteger
black	|	 esriFieldTypeInteger
class	|	 esriFieldTypeString
females	|	 esriFieldTypeInteger
hawn_pl	|	 esriFieldTypeInteger
hispanic	|	 esriFieldTypeInteger
males	|	 esriFieldTypeInteger
white	|	 esriFieldTypeInteger

Adding attribute values to the new columns

Next we can loop through each row in the third csv and add the new attribute values for these newly created columns.

In [ ]:
# Run a fresh query on the feature layer so it includes the new features from
# csv2 and new columns from csv3
cities_fset2 = cities_flayer.query()
cities_features2 = cities_fset2.features

Loop through each row in the third spreadsheet, find the corresponding feature by matching the city_id value and apply the attribute values for the new fields.

In [ ]:
features_for_update = []
for city_id in cities_df_3['city_id']:
    # get the matching row from csv
    matching_row = cities_df_3.where(cities_df_3.city_id == city_id).dropna()
    
    print(str(city_id) + " Adding additional attributes for: " + matching_row['name'].values[0])
    
    # get the feature to be updated
    original_feature = [f for f in cities_features2 if f.attributes['city_id'] == city_id][0]
    feature_to_be_updated = deepcopy(original_feature)
    
    # assign the updated values
    feature_to_be_updated.attributes['class'] = matching_row['class'].values[0]
    feature_to_be_updated.attributes['white'] = int(matching_row['white'])
    feature_to_be_updated.attributes['black'] = int(matching_row['black'])
    feature_to_be_updated.attributes['ameri_es'] = int(matching_row['ameri_es'])
    feature_to_be_updated.attributes['asian'] = int(matching_row['asian'])
    feature_to_be_updated.attributes['hawn_pl'] = int(matching_row['hawn_pl'])
    feature_to_be_updated.attributes['hispanic'] = int(matching_row['hispanic'])
    feature_to_be_updated.attributes['males'] = int(matching_row['males'])
    feature_to_be_updated.attributes['females'] = int(matching_row['females'])
    
    #add this to the list of features to be updated
    features_for_update.append(feature_to_be_updated)
1 Adding additional attributes for: Honolulu
2 Adding additional attributes for: Juneau
3 Adding additional attributes for: Boise City
4 Adding additional attributes for: Olympia
5 Adding additional attributes for: Salem
6 Adding additional attributes for: Carson
7 Adding additional attributes for: Sacramento
8 Adding additional attributes for: Phoenix
9 Adding additional attributes for: Salt Lake City
10 Adding additional attributes for: Cheyenne
11 Adding additional attributes for: Denver
12 Adding additional attributes for: Santa Fe
13 Adding additional attributes for: Oklahoma City
14 Adding additional attributes for: Topeka
15 Adding additional attributes for: Lincoln
16 Adding additional attributes for: Des Moines
17 Adding additional attributes for: Jefferson City
18 Adding additional attributes for: Little Rock
19 Adding additional attributes for: Austin
20 Adding additional attributes for: Baton Rouge
21 Adding additional attributes for: Helena
22 Adding additional attributes for: Bismarck
23 Adding additional attributes for: Pierre
24 Adding additional attributes for: St. Paul
25 Adding additional attributes for: Madison
26 Adding additional attributes for: Lansing
27 Adding additional attributes for: Augusta
28 Adding additional attributes for: Montpelier
29 Adding additional attributes for: Albany
30 Adding additional attributes for: Boston
31 Adding additional attributes for: Concord
32 Adding additional attributes for: Tallahassee
33 Adding additional attributes for: Jackson
34 Adding additional attributes for: Nashville
35 Adding additional attributes for: Montgomery
36 Adding additional attributes for: Springfield
37 Adding additional attributes for: Indianapolis
38 Adding additional attributes for: Frankfort
39 Adding additional attributes for: Columbus
40 Adding additional attributes for: Charleston
41 Adding additional attributes for: Atlanta
42 Adding additional attributes for: Columbia
43 Adding additional attributes for: Raleigh
44 Adding additional attributes for: Richmond
45 Adding additional attributes for: Harrisburg
46 Adding additional attributes for: Trenton
47 Adding additional attributes for: Dover
48 Adding additional attributes for: Washington
49 Adding additional attributes for: Annapolis
50 Adding additional attributes for: Hartford
51 Adding additional attributes for: Providence
In [ ]:
# inspect one of the features
features_for_update[-1]
Out[ ]:
{"attributes": {"class": "City", "hispanic": 52146, "pop2007": 183731, "city_id": 51, "ameri_es": 1975, "state": "RI", "name": "Providence", "latitude": 41.82355466, "black": 25243, "white": 94666, "pop2000": 173618, "asian": 10432, "objectid": 51, "longitude": -71.42212709, "hawn_pl": 270, "females": 90583, "males": 83035, "capital": "State"}, "geometry": {"x": -7950674.8190312665, "y": 5134585.3226086125}}
In [ ]:
# apply the edits to the feature layer
cities_flayer.edit_features(updates= features_for_update)
Out[ ]:
{'addResults': [],
 'attachments': {'addResults': [], 'deleteResults': [], 'updateResults': []},
 'deleteResults': [],
 'updateResults': [{'globalId': None, 'objectId': 1, 'success': True},
  {'globalId': None, 'objectId': 2, 'success': True},
  {'globalId': None, 'objectId': 3, 'success': True},
  {'globalId': None, 'objectId': 4, 'success': True},
  {'globalId': None, 'objectId': 5, 'success': True},
  {'globalId': None, 'objectId': 6, 'success': True},
  {'globalId': None, 'objectId': 7, 'success': True},
  {'globalId': None, 'objectId': 8, 'success': True},
  {'globalId': None, 'objectId': 9, 'success': True},
  {'globalId': None, 'objectId': 10, 'success': True},
  {'globalId': None, 'objectId': 11, 'success': True},
  {'globalId': None, 'objectId': 12, 'success': True},
  {'globalId': None, 'objectId': 13, 'success': True},
  {'globalId': None, 'objectId': 14, 'success': True},
  {'globalId': None, 'objectId': 15, 'success': True},
  {'globalId': None, 'objectId': 16, 'success': True},
  {'globalId': None, 'objectId': 17, 'success': True},
  {'globalId': None, 'objectId': 18, 'success': True},
  {'globalId': None, 'objectId': 19, 'success': True},
  {'globalId': None, 'objectId': 20, 'success': True},
  {'globalId': None, 'objectId': 21, 'success': True},
  {'globalId': None, 'objectId': 22, 'success': True},
  {'globalId': None, 'objectId': 23, 'success': True},
  {'globalId': None, 'objectId': 24, 'success': True},
  {'globalId': None, 'objectId': 25, 'success': True},
  {'globalId': None, 'objectId': 26, 'success': True},
  {'globalId': None, 'objectId': 27, 'success': True},
  {'globalId': None, 'objectId': 28, 'success': True},
  {'globalId': None, 'objectId': 29, 'success': True},
  {'globalId': None, 'objectId': 30, 'success': True},
  {'globalId': None, 'objectId': 31, 'success': True},
  {'globalId': None, 'objectId': 32, 'success': True},
  {'globalId': None, 'objectId': 33, 'success': True},
  {'globalId': None, 'objectId': 34, 'success': True},
  {'globalId': None, 'objectId': 35, 'success': True},
  {'globalId': None, 'objectId': 36, 'success': True},
  {'globalId': None, 'objectId': 37, 'success': True},
  {'globalId': None, 'objectId': 38, 'success': True},
  {'globalId': None, 'objectId': 39, 'success': True},
  {'globalId': None, 'objectId': 40, 'success': True},
  {'globalId': None, 'objectId': 41, 'success': True},
  {'globalId': None, 'objectId': 42, 'success': True},
  {'globalId': None, 'objectId': 43, 'success': True},
  {'globalId': None, 'objectId': 44, 'success': True},
  {'globalId': None, 'objectId': 45, 'success': True},
  {'globalId': None, 'objectId': 46, 'success': True},
  {'globalId': None, 'objectId': 47, 'success': True},
  {'globalId': None, 'objectId': 48, 'success': True},
  {'globalId': None, 'objectId': 49, 'success': True},
  {'globalId': None, 'objectId': 50, 'success': True},
  {'globalId': None, 'objectId': 51, 'success': True}]}

Verify the changes made so far

Let us run another query on the feature layer and visualize a few rows.

In [ ]:
cities_fset3 = cities_flayer.query()
cities_fset3.df.head(5)
Out[ ]:
ameri_es asian black capital city_id class females hawn_pl hispanic latitude longitude males name objectid pop2000 pop2007 state white SHAPE
0 689 207588 6038 State 1 Census Designated Place 189029 25457 16229 21.305782 -157.823436 182628 Honolulu 1 371657 378587 HI 73093 {'x': -17568824.553, 'y': 2428377.352700006}
1 3496 1438 248 State 2 City and Borough 15242 116 1040 58.351418 -134.511582 15469 Juneau 2 30711 31592 AK 22969 {'x': -14973760.769500002, 'y': 8041504.674200...
2 1300 3870 1437 State 3 City 93773 302 8410 43.613736 -116.237655 92014 Boise City 3 185787 203529 ID 171204 {'x': -12939516.521100001, 'y': 5405860.248099...
3 553 2473 805 State 4 City 22195 125 1863 47.042418 -122.893073 20319 Olympia 4 42514 45523 WA 36246 {'x': -13680394.263852797, 'y': 5949000.547924...
4 2064 3304 1750 State 5 City 68172 643 19973 44.931109 -123.029155 68752 Salem 5 136924 152039 OR 113746 {'x': -13695542.842799995, 'y': 5610682.544100...

Conclusion

In this sample, we observed an edit intensive method to keep feature layers updated. We published data from first spreadsheet as a feature layer. We then updated existing features from second spread sheet (used geometry module to project the coordinates in the process), and added new features. The third spreadsheet presented additional attribute columns which were added to the feature layer by editing its definition and then updating the features with this additional data.

This method is editing intensive and you may choose this when the number of features to edit is less or if you needed to selectively update certain features as updates come in.

An alternate method is to overwrite the feature layer altogether when you always have current information coming in. This method is explained in the sample Overwriting feature layers


Feedback on this topic?