Detect incidents

Detect Incidents examines time-sequential records using a specified condition. Records that meet the condition are marked as incidents. The result contains the input DataFrame records, with additional fields stating if the record is an incident, the status of the incident, the duration of the incident, and a unique incident identifier.

di workflow

Usage notes

  • The following table outlines terminology for Detect Incidents:
TermDescription
TrackA sequence of records that are time enabled with time type instant. Records are determined to be in the sequence by a track identifier field and are ordered by time. For example, a city could have a fleet of snowplow trucks that record their location every 10 minutes. The vehicle ID could represent the distinct tracks.
IncidentRecords that meet a condition of interest.
InstantA single moment in time represented by a start time and no end time. Inputs to Detect Incidents must have a time type of instant. Learn how to enable time on a data frame.
IntervalA duration of time represented by a start and an end time.
Record of interestDescribes the record being analyzed. During analysis, all records are analyzed.
  • Detect Incidents will create a new output. It will not modify the input DataFrame.

  • Detect Incidents can be completed on DataFrames that are tabular or have a geometry. The input DataFrame must have an instant timestamp.

  • Only input records that have a time value will be used. Any record that does not have a time value will be excluded from analysis.

  • Tracks are represented by the unique combination of one or more track fields. For example, if the flightID and Destination fields are used as track identifiers, the records ID007, Solden and ID007, Tokyo would be in different tracks since they have different values for the Destination field.

  • Conditions are created using Arcade expressions. A start condition is required, and an end condition is optional. If you only apply a start condition, the incident starts when the start condition is evaluated as true and ends when the start condition is evaluated as false. For example, if values in a track are [0, 10, 15, 20, 40, 10, 12, -2, -12 ] and the start condition is $feature['values' ] > 15, the rows that are incidents are those with [True] and would be [0: False, 10: False, 15: False, 20: True, 40: True, 10: False, 12: False, -2: False, -12: False ], where only values above 15 are incidents. If you apply an end condition of $feature['values' ] < 0, the results would be [0: False, 10: False, 15: False, 20: True, 40: True, 10: True, 12: True, -2: False, -12: False ]. In this example, the incident starts when the start condition is met, and each sequential row is an incident until the end condition is true. These examples are outlined in the following table:

Position123456789
Value0101520401012-2-12
Start: $feature['values' ] > 15 and no EndFalseFalseFalseTrueTrueFalseFalseFalseFalse
Start: $feature['values' ] > 15 and End: $feature ['values' ] < 0FalseFalseFalseTrueTrueTrueTrueFalseFalse
  • Applying a time interval segments tracks at a defined interval. For example, if you set a value of 1 day using setTimeBoundarySplit(), and a value of 9:00 a.m. on January 1, 1990 using the time_boundary_reference parameter, each track will be truncated at 9:00 a.m every day. This split accelerates computing time, as it creates smaller tracks for analysis. If splitting by a recurring time interval boundary makes sense for your analysis, it is recommended for big data processing.

  • Conditions can be track aware.

  • A track can have multiple incidents.

  • The duration of an incident is calculated in milliseconds as the time of the record of interest minus the start of an incident. The duration is only calculated if the record has a status of Started, OnGoing, or Ended. The duration for a row with the status of Started is always 0.

Limitations

The input must be a time-enabled DataFrame of type instant. Any records that do not have time will not be included in the output.

Results

Results will include the fields from the input DataFrame as well as the following additional fields:

FieldDescriptionNotes
IncidentIDA unique ID given to every row that is an incident.
IncidentStatusA string field representing the status of an incident. The value will be null if the row is not an incident, Started if the row is the first incident to meet the start condition, OnGoing if the row is still an incident, and Ended when a row is no longer an incident. A single track can have multiple segments of incidents. For example, a track with values [0, 10, 15, 20, 40, 10, 12, -2, -12] and a start condition of $feature['values' ] > 15 will result in IncidentStatus values of [0: null, 10: null, 15: null, 20: Started, 40: OnGoing, 10: Ended, 12: null, -2: null, -12: null].
IncidentDurationThe length of time, in milliseconds, an incident occurs. This is calculated as the difference between the record of interest and the record that started the incident.

Performance notes

Improve the performance of Detect Incidents by doing one or more of the following:

  • Only analyze the records in your area of interest. You can pick the records of interest by using one of the following SQL functions:

    • ST_Intersection—Clip to an area of interest represented by a polygon. This will modify your input records.
    • ST_EnvIntersects—Select records that intersect an envelope.
    • ST_Intersects—Select records that intersect another dataset or area of intersect represented by a polygon.
  • Specify Incidents as the value for the output_type parameter in the setOutputMode() setter.
  • Split your tracks using setTimeBoundarySplit().

Similar capabilities

Syntax

For more details, go to the GeoAnalytics Engine API reference for detect incidents.

SetterDescriptionRequired
run(dataframe)Runs the Detect Incidents tool using the provided DataFrame.Yes
setEndConditionExpression(end_condition_expression)Sets the condition used to end incidents. If there is an end condition, any record that meets the start condition expression and does not meet the end condition expression is an incident.No
setOutputMode(output_mode)Sets which observations are returned. Choose from 'All' (default) or 'Incidents'.No
setStartConditionExpression(start_condition_expression))Sets the condition used to start incidents. If there is no end condition expression specified, any record that meets this condition is an incident. If there is an end condition, any record that meets the start condition expression and does not meet the end condition expression is an incident.Yes
setTimeBoundarySplit(time_boundary_split, time_boundary_split_unit, time_boundary_reference=None)Sets boundaries to limit calculations to defined spans of time.No
setTrackFields(*track_fields)Sets one or more fields used to identify distinct tracks.Yes

Examples

Run Detect Incidents

Python
Use dark colors for code blocksCopy
                                                                 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
# Log in
import geoanalytics
geoanalytics.auth(username="myusername", password="mypassword")

# Imports
from geoanalytics.tools import DetectIncidents
from geoanalytics.sql import functions as ST
from pyspark.sql import functions as F

# Path to the Atlantic hurricanes data
data_path = r"https://sampleserver6.arcgisonline.com/arcgis/rest/services/" \
            "Hurricanes/MapServer/0"

# Create an Atlantic hurricanes DataFrame
df = spark.read.format("feature-service").load(data_path) \
                    .st.set_time_fields("Date_Time")

# Run Detect Incidents to find where the windspeed of a hurricane is greater than
# or equal to 70 nautical miles per hour (knots)
result = DetectIncidents() \
   .setTrackFields("EVENTID") \
   .setStartConditionExpression(start_condition_expression="$feature.WINDSPEED >= 70") \
   .setOutputMode("Incidents") \
   .run(dataframe=df)

# Convert IncidentDuration from milliseconds to hours
result = result.withColumn("IncidentDuration_hours",
                           F.col("IncidentDuration") / (60 * 60 * 1000))

# Show the first 5 rows of the result DataFrame
result.select("EVENTID", "IncidentId", "IncidentStatus", "IncidentDuration",
              "IncidentDuration_hours", "Date_Time").show(5)
Result
Use dark colors for code blocksCopy
          
1
2
3
4
5
6
7
8
9
10
+-------+--------------------+--------------+----------------+----------------------+-------------------+
|EVENTID|          IncidentId|IncidentStatus|IncidentDuration|IncidentDuration_hours|          Date_Time|
+-------+--------------------+--------------+----------------+----------------------+-------------------+
|  Debby|8de8b685-d4a8-434...|       Started|               0|                   0.0|2000-08-21 13:00:00|
|  Debby|8de8b685-d4a8-434...|       OnGoing|        21600000|                   6.0|2000-08-21 19:00:00|
|  Debby|8de8b685-d4a8-434...|       OnGoing|        43200000|                  12.0|2000-08-22 01:00:00|
|  Debby|8de8b685-d4a8-434...|         Ended|        64800000|                  18.0|2000-08-22 07:00:00|
| Gordon|62bd1f1d-9276-408...|       Started|               0|                   0.0|2000-09-17 07:00:00|
+-------+--------------------+--------------+----------------+----------------------+-------------------+
only showing top 5 rows

Plot results

Python
Use dark colors for code blocksCopy
                                                                 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
# Create a world continents DataFrame for plotting
continents_path = "https://services.arcgis.com/P3ePLMYs2RVChkJx/ArcGIS/rest/services/" \
                "World_Continents/FeatureServer/0"
continents_subset_df = spark.read.format("feature-service").load(continents_path) \
                    .where("""CONTINENT = 'North America' or CONTINENT = 'South America' or
                              CONTINENT = 'Africa' or CONTINENT = 'Europe'""")

# Plot the resulting incidents for hurricane Alberto with the USA continents data
continents_subset_plot = continents_subset_df.st.plot(facecolor="lightgreen",
                                                      edgecolors="black",
                                                      alpha=0.3,
                                                      figsize=(16,10))
alberto_tracks_plot = df.where("EVENTID = 'Alberto'").st.plot(color="lightgrey",
                                                              edgecolors="darkgrey",
                                                              ax=continents_subset_plot)
result_plot = result.where("EVENTID = 'Alberto'").st.plot(cmap_values="IncidentStatus",
                                                          cmap="tab10",
                                                          is_categorical=True,
                                                          legend=True,
                                                          legend_kwds={"title": "Incident status",
                                                                       "loc" : "upper left"},
                                                          ax=alberto_tracks_plot)
result_plot.set_title("Detected incidents for hurricane Alberto where windspeed was greater \
    than or equal to 70 nautical miles per hour (knots)")
result_plot.set_xlabel("X (Meters)")
result_plot.set_xlim(left=-20000000, right=8000000)
result_plot.set_ylabel("X (Meters)");

Plotting example for a Detect Incidents result. Finding hurricanes with high wind speeds is shown.

Version table

ReleaseNotes

1.0.0

Tool introduced

Your browser is no longer supported. Please upgrade your browser for the best experience. See our browser deprecation post for more details.