Description

Using 50 year's worth of aggregated nutrient and physical ocean data from NOAA, Esri has collaborated with the USGS, the Marine Conservation Institute, NatureServe, the University of Auckland, GRID-Arendal, NOAA, Duke University, the Woods Hole Oceanographic Institution, and many other partners to classify our oceans into 37 statistically distinct ecological marine units (EMUs). Leveraging the ArcGIS Runtime SDK for Android and ArcGIS, we'll show you how this ArcGIS Android SDK mobile app can be used to explore ocean conditions locally and globally. There are a number of ocean data resources you can use to supplement the EMU story. For example, the Ocean Biogeographic Information System hosts free and open access to ocean biogeographic datasets.

Data Prep & Optimization

Most of the data in the form of maps and feature services were published by the Oceans Group at Esri. While the data provided is useful for most of the app features, we found we needed to create and publish a web map to better visualize EMU layers by ocean depth using a continuous slider. This allows the app to filter EMU Layers by ocean depth as users interact with the slider. Below we discuss how we created and published the web map to support this user experience.

Since the EMUs are made available as points, this data was first manipulated into a form that could be interpreted more intuitively and rendered more quickly. Using ArcGIS’s suite of geoprocessing tools, Python 2.7, and the ArcPy site package, these multidimensional and overlapping clusters of points were converted into polygons. This process simplified the 52+ million points to less than 1,900 polygons while maintaining geospatial accuracy and precision. The end result represents ecologically distinct clusters of points, or EMUs, as single polygon features at every depth level. Some depth levels near the surface contain each of the 37 EMUs, while others, generally as you descend deeper into the water column, shrink to include as few as five or six. Likewise, several of the EMUs occur at each of the 100 depth levels (between 0m and 5500m deep!), while a couple EMUs only exist in the top eight or nine of these depth levels.

Generating a WebMap

Ultimately we want to consume this data as a web map in the app. In order to do this we further optimized the data into a Mobile Map Package and analyzed it such that the polygon features could be overlaid over the Oceans Basemap. We evaluated four different geoprocessing workflows in order to maintain an accurate representation of the EMU points as polygons without violating the inherent topology between neighboring data points with the following goals:

  • a: Preventing data overlaps for which given depth level, only one EMU exists per location.
  • b: Preventing data gaps such that within the world’s oceanic regions where data does exist, holes should not appear between adjacent data points.

The four geoprocessing workflows we tested:

  1. Aggregate Points -> Dissolve -> Simplify Polygon -> Graphic Buffer
  2. Point to Raster -> Raster to Polygon -> Dissolve -> Smooth Polygon
  3. Point to Raster -> Raster to Polygon (simplify) -> Dissolve
  4. Point to Raster -> Raster to Poygon -> Dissolve

We then evaluated the output generated by each of the above workflows on the basis of complexity, size on disk, and general cartographic quality. While the first two these qualities can be objectively measured, evaluating the aesthetics of the output proved to be a bit more tricky. This was made easier by the fact that approaches 1 and 2 violated goal b, and in some cases, goal a as well.

Here is a visual comparison showing each of the four outputs at the first depth level:

To compare based on complexity and disk size, the number of vertices were calculated for each of the four layers in the mobile map package we created which included only the layer specified. In both cases, the data was constrained to the first depth level and included coverage of the entire globe.

Here is a comparison of those metrics:

Ultimately, in order to best satisfy the objectives above, a point-to-raster-to-polygon approach that didn’t involve attempting to either smooth or simplify the data (workflow 4) was taken.

Automate Geoprocessing

In order to automate the geoprocessing steps above we created an ArcPy script named processEMUs.py and included in the root of the emu-app module. This tool relies on an installation of ArcGIS Desktop or Pro as described here for Pro. The script produces a geodatabase with a feature class entitled Global_EMU_Polygon which we used in ArcGIS Pro to generate our Map.

Creating the Map

With the optimized features completed we are ready to prepare the features for displaying in our map. We imported the polygon features into ArcGIS Pro and symbolized according to the official colors designated for the EMUs as shown in the chart below:

<

With the symbology in place, the polygons were overlaid on top of the Oceans Basemap and a 35% transparency was applied in ArcGIS Pro. Adding this transparency helped to reveal the underlying seafloor topography, providing relevant context as well as aiding in the explanation of why certain regions of the EMUs begin disappearing as you descend deeper.

Publishing

Once the map was finalized in ArcGIS Pro, it was published as a web map to an ArcGIS organization and made public to be consumed by our EMU app. In the following sections, we describe the app developer patterns and how we make use of this web map and other EMU services.

App Developer Patterns

Now that we have reviewed the steps for data scrubbing and publishing our content as a web map, we will now step through the developer patterns used to create the Android mobile application.

Feature Services

The heart of this application lies in the rich data stores collected by NOAA and analyzed by the scientific collaboration mentioned above. Over 50 million spatial and non-spatial datapoints are hosted through AGOL feature services. Most of the read-only feature services in this application were constructed by the Esri Oceans group, published using ArcGISPro or ArcMap, and made publicly available. The EMU feature services are also used by the Ecological Marine Unit Explorer, a web version of this application. Learn more about how this data set was assembled in this Story Map.

An ArcGISTiledLayer is used to display the ocean surface EMUs on top of a ocean basemap. The tiled layer, a cached map service containing pre-generated raster tiles, represents over 670,000 features and is used instead of a FeatureLayer for performance reasons.

// Start with an ocean basemap, at a zoom level of 1
ArcGISMap map =  new ArcGISMap(Basemap.Type.OCEANS, 0, 0, 1  );

// Attach the map the MapView
mapView.setMap(map)

// Define the EMU Ocean Surface layer
ArcGISTiledLayer layer =
  new ArcGISTiledLayer("http://esri.maps.arcgis.com/home/item.html?id=d2db1dbd6d2742a38fe69506029b83ac");
// Add the operational layer to the map
map.getOperationalLayers().add(layer);

A number of other data sources are consumed in the app but not loaded in the map view. ServiceFeatureTables provide summary and detail data for given EMU layers and water columns which are displayed in the app as charts and custom graphics in views separate from the map view.

// Provision a feature table
ServiceFeatureTable serviceFeatureTable = new
  ServiceFeatureTable("http://utility.arcgis.com/usrsvcs/servers/dbb13dad900d4014b0611358602723dd/rest/services/EMU_Point_Mesh_Cluster/MapServer/0")

Querying Feature Tables

In the app, spatial and non-spatial feature tables are queried. Spatial queries are used when the user interacts with the map - the screen location is converted to a geolocation and service feature tables are queried.

// Convert a tapped screen location to a geo location
// by overriding the onSingleTapConfirmed method
// of the MapTouchListener
public boolean onSingleTapConfirmed(MotionEvent motionEvent) {
      android.graphics.Point screenPoint = new android.graphics.Point((int) motionEvent.getX(),
          (int) motionEvent.getY());
      Point geoPoint = mapView.screenToLocation(screenPoint);

Given the geo located point, a buffer is created around the point and an envelope is calculated before querying the feature table.

Polygon bufferedLocation = GeometryEngine.buffer(geoPoint, BUFFER_SIZE);
PolygonBuilder builder = new PolygonBuilder(bufferedLocation);
Envelope envelope = builder.getExtent();

Now the spatial feature table can be queried using the derived envelope.

QueryParameters queryParameters = new QueryParameters();
queryParameters.setGeometry(envelope);

// We want all the columns returned from the query
ListenableFuture<FeatureQueryResult> futureResult =
  serviceFeatureTable.queryFeaturesAsync(queryParameters, ServiceFeatureTable.QueryFeatureFields.LOAD_ALL);
futureResult.addDoneListener(new Runnable() {
      @Override public void run() {
        try{
          FeatureQueryResult fqr = futureResult.get();
          if (fqr != null){
            final Iterator<Feature> iterator = fqr.iterator();
            while (iterator.hasNext()){
              Feature feature = iterator.next();
              Geometry geometry = feature.getGeometry();
              Map<String,Object> map = feature.getAttributes();
              processResults(map);
          }else{
             handleNullResult()
          }
         }catch (Exception e){
            handleException(e);
         }
     });

Multiple spatial results returned from the query are sorted based on their geodesic distance from the tapped location.

// Establish the center point of the envelope
final Point center = envelope.getCenter();

final LinearUnit linearUnit = new LinearUnit(LinearUnitId.METERS);
final AngularUnit angularUnit = new AngularUnit(AngularUnitId.DEGREES);

// Iterate through the results
final Geometry geo = iterator.next();
final WaterColumn waterColumn = waterColumnMap.get(geo);
final Point point = (Point) geo;
final Point waterColumnPoint = new Point(point.getX(), point.getY(), center.getSpatialReference());

// Calculate the distance
final GeodeticDistanceResult geodeticDistanceResult = GeometryEngine.distanceGeodetic(center, waterColumnPoint, linearUnit, angularUnit, GeodeticCurveType.GEODESIC);
final double calculatedDistance = geodeticDistanceResult.getDistance();
waterColumn.setDistanceFrom(calculatedDistance);
waterColumnList.add(waterColumn);

// Sort water columns
Collections.sort(waterColumnList);
// Grab the closest water column
closestWaterColumn = waterColumnList.get(0);

EMU patterns at different ocean depths are visualized by moving the slider on the web map, see the Generating a WebMap section above for info on how the data was generated.

The EMU layers are queried by setting a definition expression on the FeatureLayer.

// The feature layer containing the EMU polygons by depth
mEmuByDepthLayer = new FeatureLayer(mEmuByDepthTable);

// First check if the feature data has been cached...

if (mCachedLayers.contains(depth)){
  // Data has been downloaded and cached, so just filter the layer
  mEmuByDepthLayer.setDefinitionExpression(" Depth = " + depth);

}else{
  // No local cache, so we make a call to the server
  // by setting the where clause with the given depth
  QueryParameters queryParameters = new QueryParameters();
  queryParameters.setWhereClause(" Depth = " + depth);

  // Return all the output fields
  List<String> outFields = Collections.singletonList("*");

  final ListenableFuture<FeatureQueryResult> results =
    mEmuByDepthTable.populateFromServiceAsync(queryParameters,false, outFields);

  results.addDoneListener(new Runnable() {

      @Override public void run() {
        try {
          final FeatureQueryResult fqr = results.get();

          if (fqr.iterator().hasNext()){

            // Cache the depth level so we don't download the same data again
            mCachedLayers.add(depth);

            // Set the definition expression to show only the depth of interest

            mEmuByDepthLayer.setDefinitionExpression("Depth = " + depth);

          }
        } catch (final Exception e) {
          handleError();
        }
      }
    });
  }

Non-spatial data like summary statistics and datapoints for charts are retrieved by first putting the table in FeatureRequestMode.MANUAL_CACHE and then querying by calling pouplateFromServiceAsync.

ServiceFeatureTable summaryStats = new
  ServiceFeatureTable("http://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/EMU_Summary_Table/FeatureServer/0")
summaryStats.setFeatureRequestMode(ServiceFeatureTable.FeatureRequestMode.MANUAL_CACHE);
summaryStats.loadAsync();
summaryStats.addDoneLoadingListener(new Runnable() {
  @Override public void run() {
    QueryParameters queryParameters = new QueryParameters();
    // Get all the rows in the table
    queryParameters.setWhereClause("1 = 1");
    List<String> outFields = new ArrayList<String>();
    // Get all the fields in the table
    outFields.add("*");
    ListenableFuture<FeatureQueryResult> futureResult =
          summaryStats.populateFromServiceAsync(queryParameters,true,outFields);
    processQueryForEmuStats(futureResult);
   }
  });

Source Code on GitHub
In this topic