Feature Service

A feature service is a data service that stores and hosts spatial and nonspatial data online. In a feature service, spatial datasets are feature layers and non-spatial datasets are tables. You can query, edit and analyze data in a feature service. Feature service URLs are typically in the formats shown below.

URL for a feature service layer or table hosted in ArcGIS Online: https://<host>/<uniqueID>/ArcGIS/rest/services/<serviceName>/FeatureServer/<Layer or Table ID>

URL for a feature service layer or table hosted in ArcGIS Enterprise: https://<host>/<webadaptor>/rest/services/Hosted/<serviceName>/FeatureServer/<Layer or Table ID>.

Refer to the feature service documentation for ArcGIS Online and ArcGIS Enterprise for more details.

Register a GIS

Authentication will be required when reading or saving a secured feature service hosted by ArcGIS Online or ArcGIS Enterprise. GeoAnalytics Engine supports authentication by registering a GIS (i.e. a connection to ArcGIS Online or ArcGIS Enterprise) with the register_gis function.

The default GIS in register_gis is ArcGIS Online. The following example shows registering ArcGIS Online as a GIS with a username and password for a built-in account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import geoanalytics
geoanalytics.register_gis("myGIS", username="User", password="p@ssw0rd")

To connect to ArcGIS Enterprise, specify the portal URL as shown in the following example for a built-in account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import geoanalytics
geoanalytics.register_gis("myPortal", "https://example.com/portal", username="User", password="p@ssw0rd")

Each GIS name can only be registered once. To remove a GIS, use the unregister_gis function as shown in the following example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import geoanalytics
geoanalytics.unregister_gis("myGIS")

In addition to built-in accounts, the following authentication schemes are supported:

• Web-tier authentication secured with PKI
• User authentication with OAuth 2.0

Web-tier authentication secured with PKI

To connect to a PKI protected ArcGIS Enterprise, you can use a PKCS12 formatted certificate file (for example .pfx or .p12) and password when logging into your ArcGIS Enterprise using PKI-based client certificate authentication.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
geoanalytics.register_gis("myPortal", "https://example.com/portal",
                          cert_file=r"\\path\to\mycert.pfx", cert_password="p@ssw0rd")

User authentication with OAuth 2.0

GeoAnalytics Engine supports OAuth 2.0 as an authentication method and acts as a serverless native application when using OAuth 2.0 authorization with ArcGIS Online or ArcGIS Enterprise. This mode of authorization requires a client id. If you don't have a client id, follow the steps below to obtain one by registering a new application with your GIS. Only one client id is required, so if your GIS has one already, you may use it instead of creating a new application.

1. Log in to your ArcGIS Online or ArcGIS Enterprise organization.
2. Go to the Content tab.
3. Click New item, then select Application.
4. On the New item dialog, select Other application and click Next.
5. Type in a title for the application item. Optionally, specify the folder, tags, and summary.
6. Click Save to add the item.
7. On the item details page of this newly created application, browse to the Credentials section and find the Client ID. The string listed is the value that you will pass in as the client_id parameter when registering a GIS

You can register an OAuth 2.0 protected GIS with your client id and an authorization code. To generate an authorization code, call register_gis with only the name, url, and client_id parameters specified.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
geoanalytics.register_gis("myPortal", "https://example.com/portal", client_id="drg5d5r575rfb")

1
2
IllegalArgumentException: 'authorization_code' is required when 'client_id' is provided. You can generate an
authorization code through the following URL: ...

Open the url provided in the exception in a browser (you may need to sign in) and copy the authorization code. You can now register your GIS as shown below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
geoanalytics.register_gis("myPortal", "https://example.com/portal",
                          client_id="drg5d5r575rfb", authorization_code="3464736")

Read from feature services

GeoAnalytics Engine supports loading data from feature services into a Spark DataFrame for use with any operations supported on a DataFrame in GeoAnalytics Engine or PySpark.

The following example shows the Python syntax for loading a feature service into a Spark DataFrame, where URL is the URL to a feature service layer or table. URL is not required if specified using the DataFrameReader option listed in the table below.

1
spark.read.format("feature-service").option().load(URL)

Below is a table of options that GeoAnalytics Engine supports when loading a feature service with Spark DataFrameReader.

When loading a public feature service layer or table, you can pass the URL without options 'gis' or 'token':

1
spark.read.format("feature-service").load(URL)

When loading a secured feature service layer or table, a GIS connection or token will be required. For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import geoanalytics
geoanalytics.register_gis("myGIS", username="User", password="p@ssw0rd")
spark.read.format("feature-service").option("gis", "myGIS").load(URL)

For more examples on how to load a feature service layer, refer to the Read from feature services tutorial.

Write to feature services

GeoAnalytics Engine extends Spark and supports saving data in feature services. You can serve and share feature services over the internet. You can also access, visualize, or edit the published feature services in an ArcGIS Online or Enterprise portal.

The following script shows the Python syntax of saving feature services that GeoAnalytics Engine currently supports:

1
df.write.format("feature-service").option().save()

Below is a table of options and modes that GeoAnalytics Engine supports when saving a feature service with Spark DataFrameWriter. The default mode is to create a new feature service.

When saving a feature service, either a registered GIS or a token is required. Below is an example of registering a GIS and saving a feature service.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import geoanalytics
geoanalytics.register_gis("myGIS", username="User", password="p@ssw0rd")

data = [
    ('{"x": -0.126,"y": 51.504,"m": 187.73}', 43.26),
    ('{"x": -0.126,"y": 51.504}', 97.22)
]
df = spark.createDataFrame(data, ["esri_json", "value"]) \
          .withColumn("point", ST.point_from_esri_json("esri_json", 4326))

df.write.format("feature-service") \
        .option("gis", "myGIS") \
        .option("serviceName", "esri_json_output") \
        .save()

For more examples on how to save feature services, refer to the Write to feature services tutorial.

Usage notes

• A layer or table ID must be included in the feature service URL when loading.
• Secured feature services can be read by either providing a registered GIS or a token.
• When loading a feature service layer with geometries, the geometry field will be included in the result DataFrame automatically as as a point, multipoint, line, or polygon column.
• A registered GIS or a token and Portal URL are required when saving a feature service.
• When saving a Spark DataFrame with a geometry column, a feature layer will be created. For a Spark DataFrame without a geometry column, a table will be created in the new feature service. A spatial reference is required to be set on the geometry column when saving to a feature layer.
• The generic geometry type is not supported when saving a feature service. The geometry type should be one of the following types: point, multipoint, line, or polygon.
• Any 64-bit signed integer columns (LongType in Spark) will be cast to DoubleType and written to feature service as esriFieldTypeDouble.
• Beginning with version 1.2.0, esriFieldTypeBigInteger fields can be read from feature services as LongType columns. Reading esriFieldTypeBigInteger fields is not supported in previous versions.
• The option layerId can only be used to overwrite or append to an existing layer. Use the layerName option when creating a new layer.
• The option serviceName should only be used when writing to a new feature service, and the service name needs to be unique within your ArcGIS Enterprise or ArcGIS Online organization. To overwrite or append to an existing layer, use serviceUrl.
• When providing serviceUrl, layerId, or layerName must also be specified.
• When saving a feature service, the hasZ and hasM properties will always be set to false.