Amazon EMR Serverless runs Apache Spark workloads without requiring you to provision or manage EC2 clusters. This guide describes how to configure GeoAnalytics Engine in an interactive PySpark notebook connected to an EMR Serverless application.

For cluster-based installation instructions, see Install GeoAnalytics Engine on Amazon EMR on EC2.

The table below summarizes the Amazon EMR Serverless runtimes supported by each version of GeoAnalytics Engine.

GeoAnalytics EngineAmazon EMR Serverless
2.2.x7.3.0-8.1.0

To complete this installation, you will need:

  • An active AWS subscription.
  • An EMR Studio. If you do not already have one, set up an EMR Studio. Any VPC configured for the Studio must be in the same AWS Region as the EMR studio workspace you will create later.
  • An IAM execution role for EMR Serverless. If you do not already have one, create a job runtime role. For both new and existing roles, follow Configure IAM access below.
  • GeoAnalytics Engine install files. If you have a connected GeoAnalytics Engine subscription, you can download the ArcGIS GeoAnalytics Engine distribution here after signing in. If you have a license file, follow the instructions provided with your license file to download the GeoAnalytics Engine distribution.

  • A GeoAnalytics Engine subscription, or a license file.

Prepare the workspace

  1. Sign in to the AWS Management Console and choose the AWS Region for this setup. Your EMR Studio, EMR Serverless application, and EMR studio workspace must all be in the same Region. If you are using an existing Studio or workspace, select its Region before proceeding.

  2. Create a bucket in S3 or choose an existing one to stage setup files in.

  3. Upload the main geoanalytics JAR and the geoanalytics Python ZIP to your S3 bucket. Depending on the analysis you will perform, optionally upload the following JARs:

    • geoanalytics-natives—Geocoding and network analysis.
    • geoanalytics-raster—Raster functions, tools, and data sources.
    • geoanalytics-geoenrichment—GeoEnrichment.
    • Supplementary Projection Engine data JARs, if your transformations require them.
  4. Record the S3 URI of each uploaded file. The examples below use s3://your-bucket/geoanalytics/; replace this prefix with your own location and x.x.x with your GeoAnalytics Engine version.

Create a Serverless application

  1. In the EMR console's navigation pane, select EMR Serverless. In the Manage applications pane, select Manage applications in EMR Studio, choose your existing EMR Studio, and click Manage applications.

  2. On the Serverless applications page, choose Create application.

  3. Enter an application name and select Spark as the Type.

  4. Choose a supported EMR Release version and an Architecture compatible with your GeoAnalytics Engine dependencies. Optionally, enable Serverless storage if available for the selected release.

  5. Under Application setup options, select Use custom settings.

  6. If you plan to use geocoding or network analysis tools, you may need to increase the driver and executor disk sizes under Pre-initialized capacity to accommodate downloaded datasets, extracted packages, and temporary files.

  7. Under Interactive endpoint, select Enable endpoint for EMR Studio.

  8. Under Application configuration, enter the following JSON. Replace the S3 prefix with your own location, SCALA_VERSION with the Scala version used by your Spark runtime, and x.x.x with your GeoAnalytics Engine version. Remove optional JARs you did not upload. Add any supplementary projection or H3 JARs to the comma-separated spark.jars value as needed.

    Use dark colors for code blocksCopy
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    {
      "runtimeConfiguration": [
        {
          "classification": "spark-defaults",
          "properties": {
            "spark.jars": "s3://your-bucket/geoanalytics/geoanalytics_SCALA_VERSION-x.x.x.jar,s3://your-bucket/geoanalytics/geoanalytics-natives_SCALA_VERSION-x.x.x.jar,s3://your-bucket/geoanalytics/geoanalytics-raster_SCALA_VERSION-x.x.x.jar,s3://your-bucket/geoanalytics/geoanalytics-geoenrichment_SCALA_VERSION-x.x.x.jar",
            "spark.plugins": "com.esri.geoanalytics.Plugin",
            "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
            "spark.kryo.registrator": "com.esri.geoanalytics.KryoRegistrator"
          }
        }
      ]
    }
  9. EMR Serverless blocks outbound internet access by default. For connected authorization and access to public data sources, configure Network connections on the Serverless application by selecting a VPC, subnets, and security groups that allow outbound HTTPS traffic (TCP port 443). For setup instructions, see Configuring VPC access for EMR Serverless applications.

  10. Optionally, configure the rest of the settings.

  11. Choose Create and start application.

Configure IAM access

Configure the IAM trust policy and permissions for the execution role you will select when attaching the workspace, labeled Interactive runtime role in EMR Studio. This role provides the AWS resource permissions used by your notebook workloads; for more information, see Job runtime roles for Amazon EMR Serverless. The trust policy allows EMR Serverless to assume the role, while permission policies grant access to AWS resources such as the Glue Data Catalog. This role is separate from the Studio service role.

  1. In the IAM console's navigation pane, select Roles under Access management, then select the execution role. Under Trust relationships, ensure the trusted service is emr-serverless.amazonaws.com with the action sts:AssumeRole. Restrict access using your account and application as appropriate. For example, a trust policy can allow the application's ARN as follows:

    Use dark colors for code blocksCopy
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "emr-serverless.amazonaws.com"
          },
          "Action": "sts:AssumeRole",
          "Condition": {
            "StringEquals": {
              "aws:SourceAccount": "123456789012"
            },
            "ArnEquals": {
              "aws:SourceArn": "arn:aws:emr-serverless:REGION:123456789012:/applications/APPLICATION_ID"
            }
          }
        }
      ]
    }

    Return to EMR Studio and select your application from the Serverless applications list. Copy the Application ID from the application's details page, then return to IAM and replace APPLICATION_ID, REGION, and 123456789012 in the example policy with your application ID, AWS Region, and account ID. For an existing shared role, preserve required trust statements. To allow multiple application ARNs, use a JSON array, not a comma-separated string.

  2. If the Spark session uses the AWS Glue Data Catalog, add permissions to read the default database and list its user-defined functions. Spark may also attempt to create the database if it does not exist. Under the execution role's Permissions tab, choose Add permissions → Create inline policy and add the following JSON. These are role permissions, not a trust policy.

    Use dark colors for code blocksCopy
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "AccessDefaultGlueDatabase",
          "Effect": "Allow",
          "Action": [
            "glue:GetDatabase",
            "glue:CreateDatabase"
          ],
          "Resource": [
            "arn:aws:glue:REGION:123456789012:catalog",
            "arn:aws:glue:REGION:123456789012:database/default"
          ]
        },
        {
          "Sid": "ListDefaultGlueFunctions",
          "Effect": "Allow",
          "Action": "glue:GetUserDefinedFunctions",
          "Resource": [
            "arn:aws:glue:REGION:123456789012:catalog",
            "arn:aws:glue:REGION:123456789012:database/default",
            "arn:aws:glue:REGION:123456789012:userDefinedFunction/default/*"
          ]
        }
      ]
    }

    Replace REGION and 123456789012 with your AWS Region and account ID, then save the policy. Allow time for IAM changes to propagate before running the function-listing command below.

(Optional) Check application status and view logs

  1. In EMR Studio, select your application from the Serverless applications list and review its status. If it is stopped, attaching a workspace can start it automatically.
  2. If the application does not start successfully, review its status details and resource configuration. If a Spark session fails after attachment, inspect its driver and executor logs for class-loading, S3 access, native-library, or resource-allocation errors.
  3. Configure application monitoring to retain logs in S3 or CloudWatch as needed. See Logging for EMR Serverless. Logs from a failed startup are more informative than the notebook's final session-creation error alone.

Authorize GeoAnalytics Engine

  1. Create an EMR Studio workspace to run notebook code, or open an existing one.

  2. In the EMR Studio workspace's compute panel, select EMR Serverless application, choose your application and the Interactive runtime role configured above, then choose Attach.

  3. Open or create a notebook with the PySpark kernel.

  4. Load the GeoAnalytics Engine Python ZIP before importing the module. Replace the S3 prefix with your own location and x.x.x with your GeoAnalytics Engine version:

    Use dark colors for code blocksCopy
    1
    sc.addPyFile("s3://your-bucket/geoanalytics/geoanalytics-x.x.x.zip")

    Repeat this step when starting a new notebook session.

  5. Import the geoanalytics library and authorize it using a username and password, an API key, or a license file. See Authorization for more information. For example:

    Python
    Use dark colors for code blocksCopy
    1
    2
    3
    
    import geoanalytics
    geoanalytics.auth(username="User1", password="p@ssw0rd")
  6. Try out the API by importing the SQL functions as an easy-to-use alias like ST and listing the first 20 functions in a notebook cell:

    Python
    Use dark colors for code blocksCopy
    1
    2
    3
    
    from geoanalytics.sql import functions as ST
    spark.sql("show user functions like 'ST_*'").show()

Troubleshooting

The execution role cannot be assumed

Verify that you selected the intended role, that its trust policy permits emr-serverless.amazonaws.com, and that any source-account and source-ARN conditions match the application. The Region and application ID must both match. Allow time for IAM changes to propagate, then retry attaching the workspace. S3 permissions do not fix a trust-policy mismatch.

Authorization cannot connect to ArcGIS Online

If authorization fails with the following error, the Spark driver cannot connect to ArcGIS Online:

Use dark colors for code blocksCopy
1
2
py4j.protocol.Py4JJavaError: An error occurred while calling z:com.esri.geoanalytics.PythonEnv.runJsonCommand.
: java.lang.IllegalStateException: Unable to connect to https://arcgis.com. Please check your network connection and try again.

EMR Serverless blocks outbound internet access by default. Verify that Network connections is configured on the Serverless application, not just on EMR Studio. See the Network connections step in Create a Serverless application and AWS guidance on VPC, subnet, and security group setup.

Additional Python packages are missing

Packages available in the notebook's local environment may not be installed in the remote Spark driver environment. If import matplotlib fails, the following installs it into a writable directory for the current driver's Python process. The driver must have access to the package repository. Capturing pip's output prevents it from interfering with the notebook communication channel.

Use dark colors for code blocksCopy
1
2
3
4
5
6
7
8
9
10
11
12
import subprocess
import sys
import tempfile

packages = tempfile.mkdtemp()
result = subprocess.run(
    [sys.executable, "-m", "pip", "install", "--target", packages, "matplotlib"],
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True
)
if result.returncode:
    raise RuntimeError(result.stdout)
sys.path.insert(0, packages)

This installation applies only to the current driver's Python process, not to Spark executors, and must be repeated for a new session. To preinstall packages, use a custom EMR Serverless image.

What’s next?

You can now use any SQL function, track function, raster function, or analysis tool in the geoanalytics module.

See Data sources and Using DataFrames to learn more about how to access your data from your notebook. Also see Visualize results to get started with viewing your data on a map. For examples of what else is possible with GeoAnalytics Engine, check out the sample notebooks, tutorials, and blog posts.

Your browser is no longer supported. Please upgrade your browser for the best experience. See our browser deprecation post for more details.