Skip To Content ArcGIS for Developers Sign In Dashboard

Append Features

Introduction and Overview

Any Web GIS Administrator can attest to the fundamental importance of managing hosted feature layers in ArcGIS online. Adding data en masse to existing hosted feature layers typically involved a three step process of downloading the feature layer data, merging additional edits, then overwriting the original feature layer. While that workflow suffices in many circumstances, the append() function greatly simplifies the process. It saves a ton of time if your features layers are large, eliminating the time involved in overwriting the feature layer.

Let's take a look at some example append() workflows below. This guide document refers to the feature layer you will append data into as the target, while the item you append data from is referred to as the source.

Import Libraries

In [1]:
import os
import zipfile
import datetime as dt
from copy import deepcopy

from arcgis.gis import GIS
from arcgis.features import GeoAccessor
import arcpy
Helper Function

Throughout this notebook, you'll download items from an ArcGIS Online Organizational portal. You'll subsequently publish these items with a unique timestamp to distinguish the resulting service from any other service owned by the login you employ in this guide. Service names need to be unique across an organization and the function below will add a unique timestamp to the end of item names so the service you publish is unique and owned by you.

In [2]:
def now_dt():
    return str(int(dt.datetime.now().timestamp()))

Make a GIS Connection

In [3]:
gis = GIS(profile="your_online_profile")

Append new features from File Geodatabase

This first example appends new features from a File Geodatabase into a hosted feature layer. For best performance and reducing the chance of errors when using append(), Esri strongly recommends the schema for the source file (source) to upload matches the schema of the hosted feature service layer (target).

In this first section, the schema match between a File Geodatabase item, named SubDiv_PB11_PG48_parcels, and a hosted feature service layer you will publish.

Download the Downingtown target feature layer data

The Downingtown Parcels File Geodatabase item exists in the GIS. You'll download the file geodatabase item containing the data you want to publish as a feature layer.

In [4]:
downingtown_parcels_fgdb = gis.content.search('title:Downingtown_PA_Parcels owner:api_data_owner', 
                                              item_type='File Geodatabase')[0]
downingtown_parcels_fgdb
Out[4]:
Downingtown_PA_Parcels
temporary gdb item to use for appending dataFile Geodatabase by api_data_owner
Last Modified: May 15, 2019
0 comments, 1 views

You can use Python's os module to ensure you know where the code downloads files. By default, the file downloads to the current working directory as returned by os.getcwd(). Use os.chdir(path_to_your_directory) to change the current working directory to where you want files to download. For example, you could assign variables for specific paths on your file system like below:

  • cwd = os.getcwd()
  • wd = os.path.join(cwd, "append_guide")
  • os.chdir(wd)

The rest of the guide will use the wd variable as the working directory to manage data downloads for the exercise. You can choose to follow the example, or manage downloads in a way that works best for you.

In [13]:
wd = os.getcwd()
In [14]:
if not os.path.exists(os.path.join(wd, "downingtown")):
    os.mkdir(os.path.join(wd, "downingtown"))

# Assign a variable to the full path of the new directory to use as your working directory
wd = os.path.join(wd, "downingtown")

Extract the file geodatabase. The save_path and file_name parameters will be unique to you and your system. The paths and names used in this guide are solely for demonstration purposes.

In [15]:
downingtown_zip = downingtown_parcels_fgdb.download(save_path=wd, file_name="downingtown_fgdb_"
                                                    + now_dt() + ".zip")

Publish the target Downingtown hosted feature layer

Set properties for a new item, then add it to the portal with the zip file as the data argument. Publish the item to create a hosted feature layer your login owns to work though this guide.

In [16]:
downingtown_props = {"title":"Downingtown_PA_Parcels_" + now_dt(),
                   "type":"File Geodatabase",
                   "tags":"Downingtown PA, Pennsylvania, local government, \
                   parcel management",
                   "snippet":"Parcel data for the municipality of Downingtown, \
                   Pennsyslvania in the United States.",
                   "description":"Data downloaded from Chester County, PA \
                   Open Data https://data1-chesco.opendata.arcgis.com"}

if not "Downingtown" in [folder['title'] for folder in gis.users.me.folders]:
    gis.content.create_folder("Downingtown")

downingtown_fgdb_item = gis.content.add(item_properties=downingtown_props,
                                        data=downingtown_zip,
                                        folder="Downingtown")
downingtown_item = downingtown_fgdb_item.publish()
In [17]:
downingtown_item.title
Out[17]:
'Downingtown_PA_Parcels_1559344876'

Visualize the Downingtown feature layer

In [18]:
downingtown_fl = downingtown_item.layers[0]
downingtown_fl
Out[18]:
<FeatureLayer url:"https://services7.arcgis.com/JEwYeAy2cc8qOe3o/arcgis/rest/services/Downingtown_PA_Parcels_1559344876/FeatureServer/0">
In [19]:
downingtown_fl.query(return_count_only=True)
Out[19]:
2548
In [23]:
m = gis.map("Downingtown, PA")
m.center = [40.0065, -75.7033]
m.zoom = 14
m
In [19]:
m.add_layer(downingtown_fl)

We'll zoom in to a particular part of Downingtown where a new subdivision has been build but has not been added to the feature layer. (Observe the empty square-shaped section in the map after running the cell below.

In [20]:
m.zoom = 16
m.center = [39.9975, -75.7173]

Query the Feature for a specific subdivision. Currently, there are no features for this subdivision. You will be adding the features for subdivision as PB 11 PG 48 to the parcels feature layer for Downingtown, PA using the append features workflow.

In [20]:
downingtown_fl.query(where="SUBDIV_NUM = 'PB 11 PG 48'")
Out[20]:
<FeatureSet> 0 features

Get the source Downingtown subdivision item for appending

For the append workflow to work, you need to add the source file containing the new features as an Item in your GIS. In this case this file is already uploaded as an Item for you. Next, get the item id value for this file geodatabase item containing features you want to append to the feature layer. In this case, the Schema from the file geodatabase you'll append matches the hosted feature layer.

In [23]:
down_subdiv_fgdb_item = gis.content.search(query="title:SubDiv* owner:api_data_owner", item_type="File Geodatabase")[0]
In [25]:
subdiv_id = down_subdiv_fgdb_item.id
subdiv_id
Out[25]:
'4181764cd26b4dc0bea29dda673c0c7e'

Append the features

To update the target FeatureLayer object with a new set of features, you can call the append() method on it and give it the source Item containing the new set of features.

Run append with the appropriate parameters to add the subdivision features from the file geodatabase item . Since the file geodatabase schema matches the schema of Downingtown parcels Feature Layer, only the source_table_name from the file geodatabase item is needed to insert new features.

In [26]:
downingtown_fl.append(item_id=subdiv_id, 
                      upload_format='filegdb',
                      source_table_name='subdiv_pb_11_pg_48')
Out[26]:
True

Verify the append

We see a message of True.

Query the feature layer and then visualize it to verify results.

In [27]:
downingtown_fl.query(where="SUBDIV_NUM = 'PB 11 PG 48'")
Out[27]:
<FeatureSet> 28 features

Display layer with appended Features

We'll retrieve the layer again from out portal and added it to a new map to verify the new features were added.

In [34]:
m2 = gis.map("Downingtown, PA")
m2.center = [40.0065, -75.7033]
m2.zoom = 14
m2