Append Features¶

Any Web GIS Administrator can attest to the fundamental importance of managing hosted feature layers in ArcGIS online. Adding data en masse to existing hosted feature layers typically involved a three step process of downloading the feature layer data, merging additional edits, then overwriting the original feature layer. While that workflow suffices in many circumstances, the append() function greatly simplifies the process. It saves a ton of time if your features layers are large, eliminating the time involved in overwriting the feature layer.

Let's take a look at some example append() workflows below. This guide document refers to the feature layer you will append data into as the target, while the item you append data from is referred to as the source.

Import the Libraries
Make a GIS Connection
Append new features from a File Geodatabase
Insert attribute values from a CSV
Insert new features and update (upsert) attribute values from a File Geodatabase

Import Libraries¶

In [1]:

import os
import zipfile
import datetime as dt
from copy import deepcopy

from arcgis.gis import GIS
from arcgis.features import GeoAccessor
import arcpy

Helper Function¶

Throughout this notebook, you'll download items from an ArcGIS Online Organizational portal. You'll subsequently publish these items with a unique timestamp to distinguish the resulting service from any other service owned by the login you employ in this guide. Service names need to be unique across an organization and the function below will add a unique timestamp to the end of item names so the service you publish is unique and owned by you.

In [2]:

def now_dt():
    return str(int(dt.datetime.now().timestamp()))

Make a GIS Connection¶

The cell below is a generic placeholder to illustrate connecting to your GIS through a profile holding your specific credentials. This notebook accesses data shared within a particular ArcGIS Online Organization accessible with the following credentials:

url = https://www.arcgis.com
username = arcgis_python
password = P@ssword123

So make a connection with the following syntax to continue running through the notebook: gis = GIS("https://www.arcgis.com", "arcgis_python", "P@ssword123")

In [3]:

gis = GIS('home')

Append new features from File Geodatabase¶

This first example appends new features from a File Geodatabase into a hosted feature layer. For best performance and reducing the chance of errors when using append(), Esri strongly recommends the schema for the source file (source) to upload matches the schema of the hosted feature service layer (target).

In this first section, the schema match between a File Geodatabase item, named SubDiv_PB11_PG48_parcels, and a hosted feature service layer you will publish.

Download the Downingtown target feature layer data¶

The Downingtown Parcels File Geodatabase item exists in the GIS. You'll download the file geodatabase item containing the data you want to publish as a feature layer.

In [4]:

downingtown_parcels_fgdb = gis.content.search('title:Downingtown_PA_Parcels owner:api_data_owner', 
                                              item_type='File Geodatabase')[0]
downingtown_parcels_fgdb

Out[4]:

Downingtown_PA_Parcels
temporary gdb item to use for appending data

File Geodatabase by api_data_owner
Last Modified: May 15, 2019
0 comments, 1 views

You can use Python's os module to ensure you know where the code downloads files. By default, the file downloads to the current working directory as returned by os.getcwd(). Use os.chdir(path_to_your_directory) to change the current working directory to where you want files to download. For example, you could assign variables for specific paths on your file system like below:

cwd = os.getcwd()
wd = os.path.join(cwd, "append_guide")
os.chdir(wd)

The rest of the guide will use the wd variable as the working directory to manage data downloads for the exercise. You can choose to follow the example, or manage downloads in a way that works best for you.

In [13]:

wd = os.getcwd()

In [14]:

if not os.path.exists(os.path.join(wd, "downingtown")):
    os.mkdir(os.path.join(wd, "downingtown"))

# Assign a variable to the full path of the new directory to use as your working directory
wd = os.path.join(wd, "downingtown")

Extract the file geodatabase. The save_path and file_name parameters will be unique to you and your system. The paths and names used in this guide are solely for demonstration purposes.

In [15]:

downingtown_zip = downingtown_parcels_fgdb.download(save_path=wd, file_name="downingtown_fgdb_"
                                                    + now_dt() + ".zip")

Publish the target Downingtown hosted feature layer¶

Set properties for a new item, then add it to the portal with the zip file as the data argument. Publish the item to create a hosted feature layer your login owns to work though this guide.

In [16]:

downingtown_props = {"title":"Downingtown_PA_Parcels_" + now_dt(),
                   "type":"File Geodatabase",
                   "tags":"Downingtown PA, Pennsylvania, local government, \
                   parcel management",
                   "snippet":"Parcel data for the municipality of Downingtown, \
                   Pennsyslvania in the United States.",
                   "description":"Data downloaded from Chester County, PA \
                   Open Data https://data1-chesco.opendata.arcgis.com"}

if not "Downingtown" in [folder['title'] for folder in gis.users.me.folders]:
    gis.content.create_folder("Downingtown")

downingtown_fgdb_item = gis.content.add(item_properties=downingtown_props,
                                        data=downingtown_zip,
                                        folder="Downingtown")
downingtown_item = downingtown_fgdb_item.publish()

In [17]:

downingtown_item.title

Out[17]:

'Downingtown_PA_Parcels_1559344876'

Visualize the Downingtown feature layer¶

In [18]:

downingtown_fl = downingtown_item.layers[0]
downingtown_fl

Out[18]:

<FeatureLayer url:"https://services7.arcgis.com/JEwYeAy2cc8qOe3o/arcgis/rest/services/Downingtown_PA_Parcels_1559344876/FeatureServer/0">

In [19]:

downingtown_fl.query(return_count_only=True)

Out[19]:

In [23]:

m = gis.map("Downingtown, PA")
m.center = [40.0065, -75.7033]
m.zoom = 14
m

In [19]:

m.add_layer(downingtown_fl)

We'll zoom in to a particular part of Downingtown where a new subdivision has been build but has not been added to the feature layer. (Observe the empty square-shaped section in the map after running the cell below.

In [20]:

m.zoom = 16
m.center = [39.9975, -75.7173]

Query the Feature for a specific subdivision. Currently, there are no features for this subdivision. You will be adding the features for subdivision as PB 11 PG 48 to the parcels feature layer for Downingtown, PA using the append features workflow.

In [20]:

downingtown_fl.query(where="SUBDIV_NUM = 'PB 11 PG 48'")

Out[20]:

<FeatureSet> 0 features

Get the source Downingtown subdivision item for appending¶

For the append workflow to work, you need to add the source file containing the new features as an Item in your GIS. In this case this file is already uploaded as an Item for you. Next, get the item id value for this file geodatabase item containing features you want to append to the feature layer. In this case, the Schema from the file geodatabase you'll append matches the hosted feature layer.

In [23]:

down_subdiv_fgdb_item = gis.content.search(query="title:SubDiv* owner:api_data_owner", item_type="File Geodatabase")[0]

In [25]:

subdiv_id = down_subdiv_fgdb_item.id
subdiv_id

Out[25]:

'4181764cd26b4dc0bea29dda673c0c7e'

`Append` the features¶

To update the target FeatureLayer object with a new set of features, you can call the append() method on it and give it the source Item containing the new set of features.

Run append with the appropriate parameters to add the subdivision features from the file geodatabase item . Since the file geodatabase schema matches the schema of Downingtown parcels Feature Layer, only the source_table_name from the file geodatabase item is needed to insert new features.

In [26]:

downingtown_fl.append(item_id=subdiv_id, 
                      upload_format='filegdb',
                      source_table_name='subdiv_pb_11_pg_48')

Out[26]:

True

NOTE: While technically not required for source file geodatabase items with only one stand-alone table, best practice is to treat the source_table_name parameter as required for file geodatabase source items.

Verify the append¶

We see a message of True.

Query the feature layer and then visualize it to verify results.

In [27]:

downingtown_fl.query(where="SUBDIV_NUM = 'PB 11 PG 48'")

Out[27]:

<FeatureSet> 28 features

Display layer with appended Features

We'll retrieve the layer again from out portal and added it to a new map to verify the new features were added.

In [34]:

m2 = gis.map("Downingtown, PA")
m2.center = [40.0065, -75.7033]
m2.zoom = 14
m2

ArcGIS API for Python Guide