Summarize within

Summarize Within calculates statistics in areas where geometries from the input DataFrame are within or overlap specified boundaries. The boundaries can be a polygon geometry column or you can summarize within hexagonal or square bins.

Usage notes

• Summarize Within takes a DataFrame and a boundary, and stacks them on top of each other. After stacking, you can look down through the stack and count the number of summarized input records that fall within the input boundaries. You can also calculate statistics about the attributes of the input summary records, such as minimum, maximum, sum, mean, weighted mean, and so on.

• There are two ways to specify the boundaries:

  • Use a polygon DataFrame by specifying setSummaryPolygons().
  • Use a square or hexagonal bin of a specified size that is generated when the analysis is run by specifying setSummaryBins().

• The bin size specifies how large the bins are. If you are summarizing into hexagons, the size is the height of each hexagon, and the radius of the resulting hexagon will be the height divided by the square root of three. If you are aggregating into squares, the bin size is the height of the square, which is equal to the width.

  

• Use Summarize Within to calculate standard statistics as well as geographically weighted statistics. Standard statistics summarize the statistical values without weighting. Weighted statistics calculate values using the geographically weighted attributes of lines within a polygon, or the attributes of polygons within a polygon. Weighted statistics do not apply to points within polygons.

• Standard statistics and geographically weighted statistics can be calculated for attributes that represent either counts or rates. These are defined as follows:

  • Counts—Attributes that represent a sum or quantity of an entity at a point location, along a line, or within a polygon. Examples of count-type attributes include the population of a country, the number of taxi pickups in a census block, and the number of dams along a river. For line and polygon features, counts are proportioned before calculating standard or weighted statistics.
  • Rates—Attributes that represent a ratio or index at a point location, along a line, or within a polygon. Examples of rate-type attributes include the population density of a country, the speed limit of a road, or the walkability score of a neighborhood. Rates are never proportioned.

  For count-type attributes, values are proportioned according to the amount of the line within a polygon or the amount of the polygon within another polygon prior to calculating statistics. Statistics are calculated the same way for count-type and rate-type attributes when the summary features are points.

• You can calculate the lengths and areas of the summarized geometries within each polygon using the options in the table below. Options are based on the geometry of the summarized DataFrame.

• For standard statistics, there are ten options: count, sum, mean, minimum, maximum, range, standard deviation, variance, first, and last. Count and sum will not be calculated for rate-type attributes. There are four options for string statistics: count, any, first, and last.

• For weighted statistics, there are three options: mean, standard deviation, and variance. Weighted statistics are not calculated for string data.

• To calculate first or last, time needs to be enabled on the input DataFrame.

  Learn more about enabling time on a DataFrame.

• Analysis with binning requires that your input DataFrame's geometry has a projected coordinate system. If your data is not in a projected coordinate system, the tool will transform your input to a World Cylindrical Equal Area (SRID: 54034) projection. You can transform your data to a projected coordinate system by using ST_Transform.

  Learn more about coordinate systems and transformations

• Optionally, specify a field name using setGroupBy() so statistics are calculated separately for each unique field value. When a group by field value is specified, a summary table listing each record and statistic is also created.

• The options include_minor_major_fields and include_group_percentages are part of the group by option (setGroupBy()). The minority and majority will be the least and most dominant value from the group field, respectively, where dominance is determined using the count of points, total length, or total area of each value.

• When the specified value for include_minor_major_fields is True, two fields will be added to the result DataFrame. The fields will list the values from the group field that are the minority and majority for each result.

• The include_group_percentages option can only be used when you specify a value of True for include_minor_major_fields. When the value specified for include_group_percentages is True, two fields will be added to the result DataFrame listing the percentage of the count of points, total length, or total area that belong to the minority and majority values for each input record. A percentage field will also be added to the result table listing the percentage of the count of points, total length, or total area that belong to all values from the group by field for each input record.

• The output DataFrame always contains polygons. Only polygons that intersect the summarized geometries will be returned. Other polygons will be completely removed from the result.

  

Limitations

• You can only calculate statistics on the records that intersect your boundary.

Results

The following fields are included in the output polygon DataFrame:

The following fields are included in the output group-by DataFrame:

Performance notes

Improve the performance of Summarize Within by doing one or more of the following:

• Only analyze the records in your area of interest. You can pick the records of interest by using one of the following SQL functions:

  • ST_Intersection—Clip to an area of interest represented by a polygon. This will modify your input records.
  • ST_BboxIntersects—Select records that intersect an envelope.
  • ST_EnvIntersects—Select records having an evelope that intersects the envelope of another geometry.
  • ST_Intersects—Select records that intersect another dataset or area of intersect represented by a polygon.
• If you are using bins, larger bins will perform better than smaller bins. If you are unsure which size to use, start with a larger bin to prototype.

Similar capabilities

• Spatiotemporal Join
• Aggregate Points

Syntax

For more details, go to the GeoAnalytics Engine API reference for summarize within.

Examples

Run Summarize Within

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# Log in
import geoanalytics
geoanalytics.auth(username="myusername", password="mypassword")

# Imports
from geoanalytics.tools import SummarizeWithin
from geoanalytics.tools import ReconstructTracks
from geoanalytics.sql import functions as ST

# Path to the hurricane tracks dataset
hurricanes_data_path = r"https://services2.arcgis.com/FiaPA4ga0iQKduv3/arcgis/rest/" \
            "services/IBTrACS_ALL_list_v04r00_points_1/FeatureServer/0"

# Create a hurricanes tracks DataFrame and filter to a smaller extennt of area
hurricanes_df = spark.read.format("feature-service").load(hurricanes_data_path) \
    .withColumn("bbox_intersects", ST.bbox_intersects("shape",-10512137.72, -9527997.38,3278846.39,4303954.46)) \
    .where("bbox_intersects == 'true'") \
    .where("BASIN == 'NA'")

# Use Reconstruct Tracks to create hurricane paths
rt_result = ReconstructTracks() \
            .setTrackFields("NAME") \
            .setDistanceMethod(distance_method="Planar") \
            .run(dataframe=hurricanes_df)

# Use Summarize Within to summarize hurricane tracks into bins to
#     visualize a track heat map
result = SummarizeWithin() \
           .setSummaryBins(bin_size=200, bin_size_unit="Kilometers", bin_type='hexagon') \
           .includeShapeSummary(include=True, units="Kilometers") \
           .run(dataframe=rt_result)

Plot results

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# Plot the summarized result with shorelines (continent outlines) near Florida
continents_path = "https://services.arcgis.com/P3ePLMYs2RVChkJx/ArcGIS/rest/" \
    "services/World_Continents/FeatureServer/0"
shoreline_df = spark.read.format("feature-service").load(continents_path)

result_plot = result.output.st.plot(cmap_values="COUNT",
                                    cmap="coolwarm",
                                    figsize=(14,8),
                                    legend=True,
                                    basemap="light")

shoreline_plot = shoreline_df.st.plot(edgecolors="black",
                                    alpha=0.3,
                                    figsize=(14,8),ax=result_plot)


result_plot.set_title("Hurricane track heat map near Florida and the Gulf of Mexico")
result_plot.set_xlabel("X (Meters)")
result_plot.set_ylabel("Y (Meters)");
result_plot.set_xlim(left=-10512137, right=-8027997)
result_plot.set_ylim(bottom=2688846, top=4303954)

Version table

Links to helpful information

• Visualizing results
• Using DataFrames