Append (Feature Service/Layer)

  • URL:http:// <catalog-url>/<serviceName>/FeatureServer/0/append(POST only)

Description

Online feature service supports an append API that has the ability to upsert data. It is a high performance API capable of handling large volume of data. Upsert is a combination of insert and update, driven by the featureId. Upsert is used to add new features and updating existing features at the same time. If a feature in the source data matches an existing feature in the destination layer, the existing feature is updated with the values from the source. If no match, then a new feature is created based on the values from the source.

The Append currently supports featureId of type ObjectId and GlobalId. The target layer needs to have a destination field of one of these types.

Unique Index on Source Data FeatureId

The destination objectId or globalId fields do not need to be mapped to source fields of the same type. Source featureId can come from different field type as long as the values are acceptable destination featureId value.

Example:

  • Destination globalId can be mapped to the following field types: {GlobalId, Guid, string} types.
  • ObjectId can be mapped to the following field types: {short, long}.

Note: When appending with upsert=true, the append API checks for uniqueness of the source featureId. Currently it is not acceptable to append two source rows to the same destination row. The append API would return an error if more than one source feature share the same featureId.

Enable/disable Append Capability

Feature service Append capability must be enabled in order to be used by non-admin users. Organization administrators or service owner can use the append API without adding the Append capability to the feature service. The Append needs to be added to the service capabilities only if the service owner or organization administrators needs to allow non-admin user to append data to a feature service from different sources.

Organization administrators or service owner can enable or disable the Append capability on existing feature service using the feature service updateDefinition API as shown in th example below.

Example: http://services.myserver.com/<tenant>/ArcGIS/rest/admin/services/USA/FeatureServer/updateDefinition

{"capabilities":"Query, Append"}

The Append capability does not require any editing capabilities enabled. Features will be inserted or updated when upsert=true as part of the append operation without having the administrator to add Create or Update capabilities to the feature service.

SupportsAppend

A feature service and layer returns supportsAppend metadata property. When this property is absent or is false, the Append capability cannot be added to the feature service and the feature service does not support the Append operation even for the organization administrators or feature service owner.

Example: http://services.myserver.com/<tenant>/ArcGIS/rest/services/USA/FeatureServer?f=pjson

{"supportsAppend" : true}

Sync/Append Capability

The Append capability is not currently supported if the feature service has Sync or ChangeTracking capabilities enabled and vise-versa; i.e., Sync or ChangeTracking cannot be enabled on the feature service if the service has Append capability enabled.

This Append is blocked for a couple of reasons:

  • The ESRI sync engines might not be able to support syncing the changes from the append operation. The append operation might be using a very low level database API taht might prevent the use of triggers to track the changes from the append operation. Online feature service sync engine relies on triggers to track the edit changes.
  • The volume of changes from append might be quite large for the syncing process and it will be better for the client to recreate the replica after append.

Views/Append Capability

The Append capability can be enabled on views.

Preserve FeatureId

The source featureId (ObjectId or GlobalId) are only preserved when using append with upsert=true. When upsert is false, all new reows created will have its new ObjectId or GlobalId allocated by the system.

Feature Service Metadata

The feature service metadata are updated after adding or updating data using the append API. The extent and lastEditDate of the feature service will be updated.

Spatial Reference/Append Projection

The source data are projected to the layer spatial reference when appending data to the destination layers. Data from CSV, Geojson, Excel are defaulted to GCS spatial reference (4326). If the source data from these data sources are in a different spatial reference than 4326, the source spatial reference can be passed in the appendSourceInfo parameter ("sourceSR" parameter).

Append/Geocoding

Append supports geocoding geometry from CSV and Excel data. Cost and billing of geocoding is the same cost and billing of geocoding during publishing data as a feature service.

Editor Tracking and Ownership-Based Access Control (OBAC)

Editor tracking is currently supported with feature service append API. The CreationDate and EditDate, Editor, Creator are populated based on the user who is calling append. Editor tracking fields are set when inserting or updating existing features. The creator and the editor fields will be set to the append user.

Service owner and administrators will bypass any OBAC set on the feature service. For non-owner, admin, if "allowsUpdateToOthers" is set to false on the feature service, the append user can only update his/her features or features owned by anonymous users.

OBAC is not currently supported and will be supported in future release. The UX is currently allowing only the owner/admin of the feature service to append data.

TimeZoneInfo (Online Feature Service 6.1)

Source data timeZoneInfo can be specified in the appendsourceInfo json object similar to the TimeZoneInfo passed in when publishing new feature service. An example of the timeZoneInfo for source date fields can be as follows:

{" dateFieldsTimeReference " : {"timeZone" : <"timezone">,"respectDaylightSaving" : true}}

Appending MultiPatch Data (Online Feature Service 6.1)

Appending/upserting data to multipatch feature service is supported. In online feature service 6.1, the client can turn on Append capability on multipatch feature service using the feature service updateDefinition admin API. Similar to non-multipatch feature service, ObjectId, GlobalId, or any other fields with unique index can be used to upsert data to multipatch feature service.

The Append API would return an error when a client tries to append non-multipatch geometry data into a feature service that stores multipatch geometries.

Request Parameters

ParameterDetails
sourceTableName

Description: Required only when the source data contains more than one table, e.g., for filegdb.

Example: sourceTableName = "Building"

fieldMappings

Description: (Optional) Used to map source data to a destination layer. The Append default behavior is to match by name ; i.e. we try to find fields in the source data that match the destination fields.

fieldMappings parameter can be used as an input to define mappings for fields in the source that do not match the destination layer fields. There is no need to define mappings for fields that match the destination layer fields. The default behavior is to match fields by name even if they are not specified in the field mapping parameter. To prevent the default mapping fields behavior, appendFields parameter can be used with append to restrict the list of fields that will be updated or inserted when using append API.

Syntax: fieldMappings=[{"name" : <"targetName">, "sourceName" : <"sourceName">}, ...]

Example: fieldMappings=[{"source":"FACILITYID","name":"FACILITYID"},{"source":"FLOW","name":"FLOW"},{"source":"DESC","name":"DESC"}]

edits

Description: Only feature collection json is supported. Append supports all format through the uploadid.

appendSourceInfo

Description: This is only needed when appending data from Excel or CSV. The appendsourceInfo can be the publishing parameter returned from analyze the csv or excel file. Appending data from CSV or Excel file requires appendsourceInfo parameter that identifies locationType of the geometry in addition to how to parse the content of the CSV and Excel files.

The appendsourceInfo parameter is designed to be the same as analyze publishing parameters returned from analyzing CSV or Excel files.

Example: {"type":"csv","name":"StarbucksClosures","useBulkInserts":true,"sourceUrl":"","locationType":"coordinates","maxRecordCount":1000,"latitudeFieldName":"y","longitudeFieldName":"x","columnDelimiter":",","qualifier":"\"","sourceSR":{"wkid":4326,"latestWkid":4326},"targetSR":{"wkid":102100,"latestWkid":3857},"editorTrackingInfo":{"enableEditorTracking":false,"enableOwnershipAccessControl":false,"allowOthersToQuery":true,"allowOthersToUpdate":true,"allowOthersToDelete":false,"allowAnonymousToUpdate":true,"allowAnonymousToDelete":true},"layerInfo":{"currentVersion":10.51,"id":0,"name":"","type":"Table","displayField":"","description":"","copyrightText":"","defaultVisibility":true,"relationships":[],"isDataVersioned":false,"supportsAppend":true,"supportsCalculate":true,"supportsTruncate":false,"supportsAttachmentsByUploadId":true,"supportsAttachmentsResizing":true,"supportsRollbackOnFailureParameter":true,"supportsStatistics":true,"supportsAdvancedQueries":true,"supportsValidateSql":true,"supportsCoordinatesQuantization":true,"supportsApplyEditsWithGlobalIds":false,"advancedQueryCapabilities":{"supportsPagination":true,"supportsPaginationOnAggregatedQueries":true,"supportsQueryRelatedPagination":true,"supportsQueryWithDistance":true,"supportsReturningQueryExtent":true,"supportsStatistics":true,"supportsOrderBy":true,"supportsDistinct":true,"supportsQueryWithResultType":true,"supportsSqlExpression":true,"supportsAdvancedQueryRelated":true,"supportsCountDistinct":true,"supportsReturningGeometryCentroid":false,"supportsQueryWithDatumTransformation":true,"supportsHavingClause":true,"supportsOutFieldSQLExpression":true},"useStandardizedQueries":false,"geometryType":"esriGeometryPoint","allowGeometryUpdates":true,"hasAttachments":false,"htmlPopupType":"","hasM":false,"hasZ":false,"globalIdField":"","typeIdField":"","fields":[{"name":"x","type":"esriFieldTypeDouble","alias":"x","sqlType":"sqlTypeDouble","nullable":true,"editable":true,"domain":null,"defaultValue":null,"locationType":"longitude"},{"name":"city","type":"esriFieldTypeString","alias":"city","sqlType":"sqlTypeNVarchar","length":256,"nullable":true,"editable":true,"domain":null,"defaultValue":null,"locationType":"city"},{"name":"y","type":"esriFieldTypeDouble","alias":"y","sqlType":"sqlTypeDouble","nullable":true,"editable":true,"domain":null,"defaultValue":null,"locationType":"latitude"},{"name":"matchaddr","type":"esriFieldTypeString","alias":"matchaddr","sqlType":"sqlTypeNVarchar","length":256,"nullable":true,"editable":true,"domain":null,"defaultValue":null,"locationType":"unknown"},{"name":"street","type":"esriFieldTypeString","alias":"street","sqlType":"sqlTypeNVarchar","length":256,"nullable":true,"editable":true,"domain":null,"defaultValue":null,"locationType":"address"},{"name":"score","type":"esriFieldTypeInteger","alias":"score","sqlType":"sqlTypeInteger","nullable":true,"editable":true,"domain":null,"defaultValue":null,"locationType":"unknown"},{"name":"locname","type":"esriFieldTypeString","alias":"locname","sqlType":"sqlTypeNVarchar","length":256,"nullable":true,"editable":true,"domain":null,"defaultValue":null,"locationType":"unknown"},{"name":"st","type":"esriFieldTypeString","alias":"st","sqlType":"sqlTypeNVarchar","length":256,"nullable":true,"editable":true,"domain":null,"defaultValue":null,"locationType":"state"},{"name":"no","type":"esriFieldTypeInteger","alias":"no","sqlType":"sqlTypeInteger","nullable":true,"editable":true,"domain":null,"defaultValue":null,"locationType":"unknown"}],"indexes":[],"types":[],"templates":[{"name":"New Feature","description":"","drawingTool":"esriFeatureEditToolPoint","prototype":{"attributes":{"x":null,"city":null,"y":null,"matchaddr":null,"street":null,"score":null,"locname":null,"st":null,"no":null}}}],"supportedQueryFormats":"JSON, geoJSON","hasStaticData":false,"maxRecordCount":-1,"standardMaxRecordCount":32000,"tileMaxRecordCount":8000,"maxRecordCountFactor":1,"capabilities":"Create,Delete,Query,Update,Editing"}}or":1,"capabilities":"Create,Delete,Query,Update,Editing"}}

upsert

Description: (Optional) Used to specify whether the edits needs to be applied as updates if the feature already exists. The default is value is false.

Values: true | false

skipInserts

Description: (Optional) Used only when upsert is true. Used to skip inserts if the value is true. The default value is false.

Values: true | false

skipUpdates

Description: (Optional) Used only when upsert is true. Used to skip updates if the value is true. The default value is false.

Values: true | false

useGlobalIds

Description: (Optional) Used to specify whether upsert needs to use Globalid when matching features. The default value is false and Objectid is used by default.

Values: true | false

updateGeometry

Description: (Optional) Used only when upsert is true. Skip updating the geometry and update only the attributes for existing features if they match source features by objectid or globalid (as specified by useGlobalIds parameter). The default value is true.

Values: true | false

appendFields

Description: The list of destination fields to append to. This is supported when upsert=true|false.

Values: ["fieldName1", "fieldName2", ...]

Example: ["FACILITYID","FLOW","DESC"]

upsertMatchingField

Description: The layer field to be used when matching features with upsert. ObjectId, GlobalId, and any other field that has a unique index can be used with upsert.

This parameter overrides useGlobalIds; e.g., specifying upsertMatchingField will be used even if you specify useGlobalIds = true.

Example: "upsertMatchingField","MyfieldWithUniqueIndex"

appendUploadId

Description: The ID for the uploaded item that contains the source file. Used in conjunction with editUploadFormat.

Example: appendUploadId=0c6b928f590f49ebac04761bab413e49

UploadId can be passed to append API if analyzing the data is not needed. Any upload items are temporary items and will be cleaned by the system. The feature service uploads API supports single upload item to 10M and multi-part uploads for more then 10M size file.

Example: http://services.myserver.com/<tenant>/ArcGIS/rest/services/USA/FeatureServer/uploads

appendItemId

Description: The ID for the geowarehouse item that contains the source file. Used in conjunction with editsUploadFormat.

Example: appendItemId=894d8c12438v4baaac164b636f7e1e2f

The source file can be added as an item to ArcGIS portal and the itermId can be passed to the append API. The itemId can be used to analyze the source data file to know information about source fields name and provide a UX to map the source fields to the destination layer fields. The item added using portal addItem API is permeant item and would need to be cleaned up (if needed) by the caller after the append operation is finished.

Example: https://portal server Url/sharing/rest/content/users/userName/addItem

appendUploadFormat

Description: The source append data format. The default is featureCollection format.

Values: sqllite | shapefile | filegdb | featureCollection | geojson | csv | excel

rollbackOnFailure

Description: (Optional) Used to specify whether the upsert edits needs to be rolled back in case of failure. The default value is false.

Value: true | false

f

Description: The response format. The default value is html.

Value: json | html

Example Usage

Example 1 (Append layer level): Append data from shape file to layer.

http://services.myserver.com/<tenant>/ArcGIS/rest/services/USA/FeatureServer/0/append?

  • upsert: false
  • skipUpdate: false
  • useGlobalIds: false
  • updateGeometry: true
  • appendUploadId: 0c6b928f590f49ebac04761bab413e49
  • appendUploadformat: shapefile
  • rollbackOnFailure: false
  • f: json

JSON Response Example

{

"status": "processing",

"statusMessage": "Job Status for jobId: b62e9db7-507c-443d-3473-8a1f7a7e9fac",

"itemId": "cc7ddbc1e33440688d3110c885fa0b30"

}

Example 2 (Check job status): http://services.myserver.com/<tenant>/ArcGIS/rest/services/USA/FeatureServer/0/ append/jobs/b62e9db7-507c-443d-3473-8a1f7a7e9fac?

JSON Response Example

{

"layerName": "CITIES",

"submissionTime": 1520876908117,

"lastUpdatedTime": 1520876913647,

"recordCount": 2,

"status": "Completed"

}

Example 3 (Append layer level): http://services.myserver.com/<tenant>/ArcGIS/rest/services/USA/FeatureServer/0/append?

  • sourceTableName: Hydrant
  • fieldMappings: [{"source":"FACILITYID","name":"FACILITYID"},{"source":"FLOW","name":"FLOW"},{"source":"LOCDESC","name":"LOCDESC"}]
  • upsert: false
  • skipInserts: false
  • skipUpdates: false
  • useGlobalIds: false
  • updateGeometry: true
  • appendFields: ["FACILITYID","FLOW","LOCDESC"]
  • appendUploadId: 0c6b928f590f49ebac04761bab413e49
  • appendUploadformat: filegdb
  • rollbackOnFailure: true
  • f: html

JSON Response Example

{

"status": "processing",

"statusMessage": "Job Status for jobId: feeahh1e-e32c-45bf-680c-f4ed70569081",

"itemId": "aa7gdww1e55200527d3110c634fa0b30"

}

Example 4 (Check job status): http://services.myserver.com/<tenant>/ArcGIS/rest/services/USA/FeatureServer/0/ append/jobs/feeahh1e-e32c-45bf-680c-f4ed70569081?

JSON Response Example

{

"layerName": "Hydrant",

"submissionTime": 1520876908117,

"lastUpdatedTime": 1520876913647,

"recordCount": 5,

"status": "Completed"

}