Cloud Raster Format (CRF) is a raster storage format optimized for writing and reading large raster datasets. CRF datasets are stored as directories and can be accessed from locations available to Spark, such as local storage, distributed file systems, or cloud storage.
GeoAnalytics Engine supports loading CRF datasets into Spark DataFrames and writing Spark DataFrames that contain raster columns to CRF datasets. After loading CRF data into a DataFrame, you can use raster functions to perform analysis and visualize the data.
Read CRF datasets
The following example shows the Python syntax for loading a CRF dataset into a Spark DataFrame, where PATH is the path
to a CRF dataset directory.
spark.read.format("crf").option().load(PATH)Read a CRF dataset
The following examples demonstrate how to load a CRF dataset into a Spark DataFrame using both Python and Scala.
df = spark.read.format("crf").load("path/to/your/crf_directory")Below is a table of options that GeoAnalytics Engine supports when loading a CRF dataset with Spark Data.
These options are similar to the options used when reading raster files and image services.
| DataFrameReader option | Example | Description |
|---|---|---|
max | .option("max | Defines the maximum size in bytes (for example, "256m" or "1g") for each Spark partition when reading a CRF dataset. If not specified, the value from spark.sql.files.max will be used, which has a default of "128m". |
extent | .option("extent", "0, -10, 10, -5") | Filters the CRF dataset by the spatial extent specified using the format ". Values should use the units and coordinate system of the CRF dataset's original spatial reference. |
bands | .option("bands", 2) | Reads only the specified bands from the CRF dataset. |
tile | .option("tile | Defines the number of columns in each raster tile. The default is 1024. |
tile | .option("tile | Defines the number of rows in each raster tile. The default is 1024. |
sr | .option("sr", 4326) | The spatial reference that the CRF dataset will be transformed to. |
Read a CRF dataset with options
The following examples demonstrate how to load a CRF dataset using a maximum partition size, spatial extent, selected bands, tile size, and a target spatial reference.
df = spark.read.format("crf") \
.option("maxPartitionBytes", "256m") \
.option("extent", "0, -10, 10, -5") \
.option("bands", 2) \
.option("tileColumns", 2048) \
.option("tileRows", 2048) \
.option("sr", 4326) \
.load("path/to/your/crf_directory")Write CRF datasets
GeoAnalytics Engine supports writing Spark DataFrames that contain a raster column to CRF datasets. The following example
shows the Python syntax for writing a CRF dataset, where PATH is the output CRF dataset directory.
df.write.format("crf").save(PATH)Write a CRF dataset
The following examples demonstrate how to write a Spark DataFrame to a CRF dataset using both Python and Scala.
df.write.format("crf").save("path/to/output/crf_directory")Below is a table of options that GeoAnalytics Engine supports when writing a CRF dataset with Spark Data.
| DataFrameWriter option | Example | Description |
|---|---|---|
field | .option("field | Defines the raster column to write when there are multiple raster columns. If not specified, the first raster column found in the DataFrame will be used. |
sr | .option("sr", 4326) | The spatial reference that the output CRF dataset will be transformed to. If not specified, the spatial reference from the input rasters is used when available. |
max | .option("max | Defines the maximum error tolerance used when writing CRF tiles. If not specified, the write operation uses a lossy default value of 0.00001 for Float32 and Float64 rasters, and 0.5 for other pixel types. |
Write a CRF dataset with options
The following examples demonstrate how to write a CRF dataset using a raster field name, target spatial reference, and maximum error tolerance.
df.write.format("crf") \
.option("fieldName", "raster") \
.option("sr", 4326) \
.option("maxError", 0.1) \
.save("path/to/output/crf_directory")Usage notes
- CRF datasets are loaded as value rasters.
- CRF reads and writes use the data source V2 API
by default. To opt in to the data source V1 API for CRF reads and writes, add
CRFto thespark.sql.sources.useSpark configuration property before reading or writing the CRF dataset.V1 Source List
- Multidimensional CRF layers are not supported. The CRF data source supports reading and writing single-layer CRF datasets only.
- The
extentoption is evaluated against the CRF dataset's original spatial reference. If you also specify thesroption, provideextentvalues in the CRF dataset's original coordinate system. - The
tileandColumns tileoptions determine how many DataFrame rows (tiles) will be created from the CRF dataset. Higher values will create fewer DataFrame rows and larger tiles.Rows - CRF write supports Spark's default error-if-exists save behavior. The output path must not already exist. Overwrite and append modes are not supported.
- If
maxis not specified when writing CRF, the output uses the default lossy maximum error tolerance for the raster pixel type.Error - When writing rasters with non-square cells to CRF, cells are resampled to a square cell size.
- When working with large CRF datasets and planning to perform a spatial transformation, it is recommended to specify the
sr(spatial reference) option directly in the CRF read operation, rather than applying theRTfunction after reading the CRF dataset. This approach is preferred because reading large raster datasets typically involves tiling, and performing the transformation afterward can introduce slight gaps or missing pixels at tile boundaries. By specifying the desired spatial reference during the read, you ensure that the transformation is handled efficiently and accurately, minimizing the risk of data loss or visual artifacts._Transform