A parquet feature layer is a type of hosted layer in ArcGIS Online that provides fast visualization and client-side querying capabilities. Parquet feature layers are best suited for large, read-only datasets where scalability, efficient storage, and optimized visualization performance are critical. These layers are highly effective when dealing with very large datasets because they offer fast drawing times like vector tile layers but provide the customization and attribute support of a feature layer.
Read from parquet feature layers
GeoAnalytics Engine supports loading data from parquet feature layers into a Spark DataFrame for use with any operations supported on a DataFrame in GeoAnalytics Engine or Spark.
Below is a table of options that GeoAnalytics Engine supports when loading a parquet feature layer with Spark Data.
One of item, item, or path is required.
| DataFrameReader option | Example | Description |
|---|---|---|
gis | .option("gis", "my | Reference to a registered GIS. |
item | .option("item | The ID of the parquet feature layer item to read. Requires that gis is also specified. |
item | .option("item | The title of the parquet feature layer item to read. Requires that gis is also specified. |
path | .option("path", "https | The URL of a parquet feature layer item. Required if gis is not specified. |
When loading a secured parquet feature layer, a GIS connection is required. For example:
# Register a GIS to authenticate with ArcGIS Online
geoanalytics.register_gis("myGIS", username="User", password="p@ssw0rd")
# Read from a parquet feature layer item called "MyExampleLayer"
spark.read.format("parquet-feature-layer").options(gis="myGIS", itemTitle="MyExampleLayer").load()Write to parquet feature layers
GeoAnalytics Engine supports saving data in parquet feature layers hosted in ArcGIS Online.
Below is a table of options that GeoAnalytics Engine supports when saving a parquet feature layer with
Spark Data. item is required when writing a new parquet feature layer, while one of item, item, or path is required when overwriting a parquet feature layer.
| DataFrameWriter option | Example | Description |
|---|---|---|
allow | .option("allow | Whether or not to allow creating a parquet feature layer with an item title that already exists. The default is False. |
description | .option("description", "my gis data") | Item description for a new parquet feature layer. |
gis | .option("gis", "my | Reference to a registered GIS. This is always required. |
item | .option("item | The ID of a parquet feature layer item to overwrite. Not supported when creating a new parquet feature layer. |
item | .option("item | The title of the parquet feature layer item to create or overwrite. |
num | .option("num | The number of parquet files to write to when creating or overwriting a parquet feature layer. More files generally improves write performance while fewer files generally improves visualization performance. The default is 4. |
path | .option("path", "https | The URL of a parquet feature layer item to overwrite. |
snippet | .option("snippet", "example") | Item snippet for a new parquet feature layer. |
tags | .option("tags", "parquet") | Item tags for a new parquet feature layer. |
Below is an example of registering a GIS and saving a parquet feature layer.
geoanalytics.register_gis("myGIS", username="User", password="p@ssw0rd")
# Read in sample data from a public feature service
data_url = r"https://sampleserver6.arcgisonline.com/arcgis/rest/services/" \
"Earthquakes_Since1970/FeatureServer/0"
df = spark.read.format("feature-service").load(data_url)
# Write to a new parquet feature layer called "Earthquakes" using 3 parquet files
df.write.format("parquet-feature-layer").options(gis="myGIS", itemTitle="Earthquakes", numOutputFiles=3).save()Usage notes
- Columns with a generic geometry type are not supported.
- Append mode is not supported.
- When reading from an unsecured parquet feature layer, use
pathto specify the layer to read. UsingitemorId itemrequires thatTitle gisis also specified. - Parquet feature layers store geometries in WGS 1984 (WKID 4326). If your geometries have a different spatial reference, they will be automatically transformed to WGS 1984 when writing to a parquet feature layer.
- If a column created with
st.withalready exists in a DataFrame it will be replaced when writing to a parquet feature layer._geodisplay() - Z-values and m-values are not supported and will be ignored when writing to parquet feature layers.
- Time only and timestamp offset field types are not supported.