Append Features¶
Any Web GIS Administrator can attest to the fundamental importance of managing hosted feature layers in ArcGIS online. Adding data en masse to existing hosted feature layers typically involved a three step process of downloading the feature layer data, merging additional edits, then overwriting the original feature layer. While that workflow suffices in many circumstances, the append()
function greatly simplifies the process. It saves a ton of time if your features layers are large, eliminating the time involved in overwriting the feature layer.
Let's take a look at some example append()
workflows below. This guide document refers to the feature layer you will append data into as the target
, while the item you append data from is referred to as the source
.
- Import the Libraries
- Make a GIS Connection
- Append new features from a File Geodatabase
- Insert attribute values from a CSV
- Download the source
csv
data - Add source
csv
item to the GIS - Analyze the source
csv
item - Download the target feature layer data
- Manage the target feature layer schema
- Publish the target feature layer
- Get the target feature layer
- Add an attribute field to the target feature layer
- Add a unique index to the new attribute field
- Append values from the source
csv
to the target feature layer
- Download the source
- Insert new features and update (
upsert
) attribute values from a File Geodatabase
Import Libraries¶
import os
import zipfile
import datetime as dt
from copy import deepcopy
from arcgis.gis import GIS
from arcgis.features import GeoAccessor
import arcpy
Helper Function¶
Throughout this notebook, you'll download items from an ArcGIS Online Organizational portal. You'll subsequently publish these items with a unique timestamp to distinguish the resulting service from any other service owned by the login you employ in this guide. Service names need to be unique across an organization and the function below will add a unique timestamp to the end of item names so the service you publish is unique and owned by you.
def now_dt():
return str(int(dt.datetime.now().timestamp()))
Make a GIS Connection¶
The cell below is a generic placeholder to illustrate connecting to your GIS through a profile holding your specific credentials. This notebook accesses data shared within a particular ArcGIS Online Organization accessible with the following credentials:
- url = https://www.arcgis.com
- username = arcgis_python
- password = P@ssword123
So make a connection with the following syntax to continue running through the notebook:
gis = GIS("https://www.arcgis.com", "arcgis_python", "P@ssword123")
gis = GIS('home')
Append new features from File Geodatabase¶
This first example appends new features from a File Geodatabase into a hosted feature layer. For best performance and reducing the chance of errors when using append()
, Esri strongly recommends the schema for the source file (source) to upload matches the schema of the hosted feature service layer (target).
In this first section, the schema match between a File Geodatabase item, named SubDiv_PB11_PG48_parcels
, and a hosted feature service layer you will publish.
Download the Downingtown target feature layer data¶
The Downingtown Parcels File Geodatabase item exists in the GIS. You'll download the file geodatabase item containing the data you want to publish as a feature layer.
downingtown_parcels_fgdb = gis.content.search('title:Downingtown_PA_Parcels owner:api_data_owner',
item_type='File Geodatabase')[0]
downingtown_parcels_fgdb
You can use Python's os
module to ensure you know where the code downloads files. By default, the file downloads to the current working directory as returned by os.getcwd()
. Use os.chdir(path_to_your_directory)
to change the current working directory to where you want files to download. For example, you could assign variables for specific paths on your file system like below:
cwd = os.getcwd()
wd = os.path.join(cwd, "append_guide")
os.chdir(wd)
The rest of the guide will use the wd
variable as the working directory to manage data downloads for the exercise. You can choose to follow the example, or manage downloads in a way that works best for you.
wd = os.getcwd()
if not os.path.exists(os.path.join(wd, "downingtown")):
os.mkdir(os.path.join(wd, "downingtown"))
# Assign a variable to the full path of the new directory to use as your working directory
wd = os.path.join(wd, "downingtown")
Extract the file geodatabase. The save_path
and file_name
parameters will be unique to you and your system. The paths and names used in this guide are solely for demonstration purposes.
downingtown_zip = downingtown_parcels_fgdb.download(save_path=wd, file_name="downingtown_fgdb_"
+ now_dt() + ".zip")
Publish the target Downingtown hosted feature layer¶
Set properties for a new item, then add it to the portal with the zip file as the data
argument. Publish the item to create a hosted feature layer your login owns to work though this guide.
downingtown_props = {"title":"Downingtown_PA_Parcels_" + now_dt(),
"type":"File Geodatabase",
"tags":"Downingtown PA, Pennsylvania, local government, \
parcel management",
"snippet":"Parcel data for the municipality of Downingtown, \
Pennsyslvania in the United States.",
"description":"Data downloaded from Chester County, PA \
Open Data https://data1-chesco.opendata.arcgis.com"}
if not "Downingtown" in [folder['title'] for folder in gis.users.me.folders]:
gis.content.create_folder("Downingtown")
downingtown_fgdb_item = gis.content.add(item_properties=downingtown_props,
data=downingtown_zip,
folder="Downingtown")
downingtown_item = downingtown_fgdb_item.publish()
downingtown_item.title
Visualize the Downingtown feature layer¶
downingtown_fl = downingtown_item.layers[0]
downingtown_fl
downingtown_fl.query(return_count_only=True)
m = gis.map("Downingtown, PA")
m.center = [40.0065, -75.7033]
m.zoom = 14
m
m.add_layer(downingtown_fl)
We'll zoom in to a particular part of Downingtown where a new subdivision has been build but has not been added to the feature layer. (Observe the empty square-shaped section in the map after running the cell below.
m.zoom = 16
m.center = [39.9975, -75.7173]
Query the Feature for a specific subdivision. Currently, there are no features for this subdivision. You will be adding the features for subdivision as PB 11 PG 48 to the parcels feature layer for Downingtown, PA using the append features workflow.
downingtown_fl.query(where="SUBDIV_NUM = 'PB 11 PG 48'")
Get the source Downingtown subdivision item for appending¶
For the append workflow to work, you need to add the source file containing the new features as an Item in your GIS. In this case this file is already uploaded as an Item for you. Next, get the item id value for this file geodatabase item containing features you want to append to the feature layer. In this case, the Schema from the file geodatabase you'll append matches the hosted feature layer.
down_subdiv_fgdb_item = gis.content.search(query="title:SubDiv* owner:api_data_owner", item_type="File Geodatabase")[0]
subdiv_id = down_subdiv_fgdb_item.id
subdiv_id
Append
the features¶
To update the target FeatureLayer
object with a new set of features, you can call the append()
method on it and give it the source Item
containing the new set of features.
Run append
with the appropriate parameters to add the subdivision features from the file geodatabase item . Since the file geodatabase schema matches the schema of Downingtown parcels Feature Layer, only the source_table_name
from the file geodatabase item is needed to insert new features.
downingtown_fl.append(item_id=subdiv_id,
upload_format='filegdb',
source_table_name='subdiv_pb_11_pg_48')
NOTE: While technically not required for source file geodatabase items with only one stand-alone table, best practice is to treat the
source_table_name
parameter as required for file geodatabase source items.
Verify the append¶
We see a message of True
.
Query the feature layer and then visualize it to verify results.
downingtown_fl.query(where="SUBDIV_NUM = 'PB 11 PG 48'")
Display layer with appended Features
We'll retrieve the layer again from out portal and added it to a new map to verify the new features were added.
m2 = gis.map("Downingtown, PA")
m2.center = [40.0065, -75.7033]
m2.zoom = 14
m2