Create a Custom Data Feed Provider

A custom data provider connects to and fetches data from remote data sources. These data sources may be hosted static files, such as a CSV, remote APIs, or SQL or NoSQL databases. You can have one or more custom data providers in a custom data app. The custom data providers can adhere to two broadly defined types:

• Full-fetch providers are designed to pull an entire dataset from a remote data source and perform post-processing functions after all of the data have been retrieved. This type of provider may be used where:

  • datasets are small and the entire dataset can be fetched
  • the remote API does not support filtering or queries
  • the remote API is slow to respond

  All queries and filtering performed in your ArcGIS client will only be performed on the dataset that is currently loaded into memory.

• Pass-through providers allow for the remote API to perform some post-processing. By providing the analogous GeoServices API query parameters to the remote API, the remote API can return a subset of the data. This type of provider may be used where:

  • datasets are large and the entire dataset cannot be fetched
  • the remote API does support filtering or queries
  • the remote API is quick to respond

  Queries and filters in your ArcGIS client are translated in your custom data provider code into a query that your remote data source can understand. The request is sent to this remote data source, and the response is used to update what is currently in memory in your ArcGIS client. See the use of query parameters at the bottom of this page and in the example here: MongoDB sample code.

To create a custom data provider, open a command prompt on your development machine in your custom data app directory, and run the cdf createprovider <providername> command. The command will create a project template for a custom data provider in the providers folder inside the custom data app.

Understanding the Model Class

The model.js file contains a Model class for fetching data from your data sources. The Model field in the index.js file must point to this class. For each provider, you must implement the getData function that includes your custom logic for fetching data and formatting it as GeoJSON. The getData function accepts two arguments: req and callback. You must use the req argument to pass a standard Express.js request object and the callback function to return data parsed as GeoJSON. The following code shows examples of using the Model class in a template model.js file:

1
2
3
4
5
6
7
8
9
10
11
12
13
// model.js
function Model() {}
Model.prototype.getData = (req, callback) => {
  // implement provider-specific logic here
  // ...
  const geojson = {
    type: "FeatureCollection",
    features: []
  };
  callback(null, geojson);
};
// export model
module.exports = Model;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// model.js
class Model {
  // implement some code here, such as a constructor with a database connection
  async getData (req, callback) {
    // implement provider-specific logic here
    // ...
    const geojson = {
      type: "FeatureCollection",
      features: []
    };
    callback(null, geojson);
  }
}
// export model
module.exports = Model;

To callback with an error, pass the error message as the first argument to the callback function as follows: callback(err)

To callback with properly-formatted GeoJSON, call the callback function as follows: callback(null, geojson)

All of the features in the GeoJSON features array must have the same geometry-type. This requirements differs from the GeoJSON specification GeoJSON specification which allows for mixed geometries. However, feature services require uniform geometries, so it is important that this rule is enforced.

Custom Data Feeds requires that the GeoJSON be decorated with a property called metadata. It should have an object as its value. This object contains information important for translating the GeoJSON to its feature service equivalent. Key GeoJSON metadata properties can be referenced here: CDF API Reference. An example of the GeoJSON metadata field is shown below.

1
2
3
geojson.metadata = {
  idField: "id"
};

If your data does not contain a field that can be used as the OBJECTID, you can leave idField undefined. In such cases Custom Data Feeds will create an OBJECTID for each feature by default. However, this is less than ideal as it impacts performance and in very rare cases may lead to OBJECTID collisions. The best practice is to have a field on the raw data that meets the requirements of idField.

Understanding Provider Configuration

Define your configurable parameters, such as API keys and database connection strings, in the default.json file found in the config folder. Once defined, you can load the config module and access the parameters in any JavaScript file in the src directory typing this line:

const config = require('../config/default.json');

Below is an example of a configuration file for a MongoDB provider.

1
2
3
4
5
6
7
{
  "mongodb": {
    "connectString": "<your connection string>",
    "db": "<your db name>",
    "collection": "<your collection name>"
  }
}

Use Route and Query Parameters to Fetch Data

You can use route parameters to build the URL to the remote data source. Each provider provides two route parameters: host and id. The code snippet below shows how to use these parameters to build a URL.

1
2
3
4
5
6
7
8
Model.prototype.getData(request, callback) {
  const {params: {host, id}} = request
  request(`https://${host}/resource${id}`, (err, res, body) => {
    if (err) return callback(err)
    // format response body as GeoJSON
    callback(null, body)
  })
}

To access the host and id parameters, set the properties.hosts field to trueand theproperties.disableIdParam field to false in the cdconfig.json file located in the root of your custom data provider directory. Although the host parameter is available to use, id is sufficient for parameterizing your provider. Custom data providers should be targeted to single, specific data sources, and we currently recommend setting properties.hosts to false unless it is critical for your custom data provider.

You can also access the query parameters for any requests through the provider. Use the query parameters to filter the remote data so that unnecessary data processing does not need to be done on the ArcGIS client end. The code snippet below shows how to access the query parameters.

1
2
3
4
5
6
7
8
Model.prototype.getData(request, callback) {
  const {params: {host, id}, query: {where}} = request
  request(`https://${host}/resource${id}?${where}`, (err, res, body) => {
    if (err) return callback(err)
    // format response body as GeoJSON
    callback(null, body)
  })
}

Note that the where parameter in Feature Service requests is SQL-like and may require parsing and careful translation before adding it to the remote API request.

For instance, if you want to query results by more than attribute, in SQL you would use an AND operator between attributes. If your provider is directly contacting a SQL database, then the query filter generated by your ArcGIS client may work with little to no modification of the where statement. However, many REST APIs use query strings that reference attributes of the collection items, such as: https://host/collection/:id?attribute1=val1&attribute2=val2.
In this example, the raw where clause generated by a filter in an ArcGIS client could not be directly passed to the API and would require some modification in the provider code. See the MongoDB sample code for an example of how a SQL-like statement is translated.