Create a Custom Data Feed Provider

A custom data provider connects to and fetches data from remote data sources. These data sources may be hosted static files, such as a CSV, remote APIs, or SQL or NoSQL databases. You can have one or more custom data providers in a custom data app. The data loading pattern in a custom data provider adheres two broadly defined types:

• The full-fetch data loading pattern pulls data from a remote data source and performs post-processing functions after all of the data have been retrieved. This type of provider may be used where:

  • datasets are small and the entire dataset can be fetched
  • the remote API does not support filtering or queries
  • the remote API is slow to respond

  All queries and filtering performed in your ArcGIS client will only be performed on the dataset that is currently loaded into memory when using the full-fetch data loading pattern.

• The pass-through data loading pattern allows for the remote API to perform some post-processing. By providing the analogous GeoServices API query parameters to the remote API, the remote API can return a targeted subset of the data. This type of provider may be used where:

  • datasets are large and the entire dataset cannot be fetched
  • the remote API does support filtering or queries
  • the remote API is quick to respond

  With the pass-through data loading pattern, queries and filters sent from your ArcGIS client are translated in your custom data provider code into a query that your remote data source can understand. The request is sent to this remote data source, and the response is used to update what is currently in memory in your ArcGIS client. See the use of query parameters at the bottom of this page and in the example here: MongoDB sample code. See the topic on Pass-through providers for more information.

To create a custom data provider, open a command prompt on your development machine in your custom data app directory, and run the cdf createprovider <name> command. The command will create a project template for a custom data provider in the providers folder inside the custom data app.

Understanding the Model Class

The model.js file contains a Model class for fetching data from your data sources. The Model field in the index.js file must point to this class. For each provider, you must implement the getData function that includes your custom logic for fetching data and formatting it as GeoJSON. The getData function accepts two arguments: req and callback. You must use the req argument to pass a standard Express.js request object, but the callback function is optional to return data parsed as GeoJSON. The syntax that uses async should not be combined with the syntax that uses callback. Using these two together is antipattern for JavaScript and can lead to issues with error handling that may lead to process termination. The following code shows examples of using the Model class in a model.js file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// model.js
function Model() {}
Model.prototype.getData = (req, callback) => { // if using 'callback', do not use 'async'
  // implement provider-specific logic here
  // ...
  const geojson = {
    type: "FeatureCollection",
    features: []
  };
  geojson.metadata = { idField: "id", maxRecordCount: 5000 }
  callback(null, geojson);
};
// export model
module.exports = Model;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// model.js
class Model {
  // implement some code here, such as a constructor with a database connection
  async getData (req) { // if using 'async', do not use 'callback' as a second argument
    // implement provider-specific logic here
    // ...
    const geojson = {
      type: "FeatureCollection",
      features: []
    };
    return {
      ...geojson,
      metadata: {
          idField: "id", // use the id field as the ObjectID
          maxRecordCount: 5000
      }
    };
  }
}
// export model
module.exports = Model;

To callback with an error, pass the error message as the first argument to the callback function as follows: callback(err)

To callback with properly-formatted GeoJSON, call the callback function as follows: callback(null, geojson)

All of the features in the GeoJSON features array must have the same geometry-type. This requirements differs from the GeoJSON specification which allows for mixed geometries. However, feature layers require uniform geometries, so it is important that this rule is enforced.

Custom Data Feeds requires that the GeoJSON be decorated with a property called metadata. It should have an object as its value. This object contains information important for translating the GeoJSON to its feature service equivalent. Key GeoJSON metadata properties can be referenced here: CDF API Reference. An example of the GeoJSON metadata field is shown below.

1
2
3
4
geojson.metadata = {
  idField: "id",
  inputCrs: 3857
};

If your data does not contain a field that can be used as an OBJECTID, then do not define the idField property in the GeoJSON metadata. In such cases, Custom Data Feeds will generate an OBJECTID field and assign a unique value to each feature. This is not the recommended approach, as it can lead to rare OBJECTID collisions and cannot be used with the pass-through pattern implementation. It can also cause issues when queries or client features (e.g., pop-ups) reference those generated IDs. The best practice is to include a stable, unique, non-null identifier in your source data and set it as the idField.

If your data is in WGS84 (WKID 4326), there is no need to specify the inputCrs because no reprojection will need to be done by custom data feeds processing and returning geoJSON. However, if your data is stored in a different CRS, you must include the WKID in the metadata to ensure correct reprojection.

Provider Configuration

Define your configurable parameters, such as API keys and database connection strings, in a JSON file of your choosing. Typically this file would be placed in the providers/provider-name/src directory. Once defined, you can "require in" the file and access the parameters in files such as the model.js file by using the lines below:

1
2
3

const config = require('./<my-provider-config>.json');
const collection = config.mongodb.collection;`

Below is an example of a configuration file for a MongoDB provider, where mongodb is the top-level key.

1
2
3
4
5
6
7
{
  "mongodb": {
    "connectString": "<your connection string>",
    "db": "<your db name>",
    "collection": "<your collection name>"
  }
}

Creating Your Own Service Parameters

Developers have the ability to define as many service parameters as is required in their custom data provider. The developer chooses a key, label, and description for each parameter. The key is used in the provider code, whereas the label and description are user-friendly text that are viewable to publishers and administrators. These service parameters are defined in cdconfig.json as shown below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
{
  "name": "my-provider",
  "arcgisVersion": "12.0.0",
  "parentServiceType": "FeatureServer",
  "customdataRuntimeVersion": "1",
  "type": "provider",
  "editingEnabled": false,
  "properties": {
    "serviceParameters": [
      {
        "label": "Doc ID",
        "key": "doc_id",
        "description": "Google Sheets document ID."
      },
      {
        "label": "Geometry Column",
        "key": "geom_column",
        "description": "Name of the column in the file that contains the feature's geometry."
      },
      {
        "label": "Data Coordinate System",
        "key": "coordinate_system",
        "description": "The WKID representing the coordinate system the remote data is stored in."
      }
    ]
  }
}

The properties.serviceParameters array is populated with objects containing label, key, and description for each service parameter. All parameters entered in this array are considered required parameters, and publishers will need to supply values for them at service creation.

It is highly recommended that your service parameters are given useful label and description values because these values will be shown in ArcGIS Enterprise clients such as Enterprise Manager and Portal for ArcGIS. Descriptive values for these attributes will make the service creation process easier for publishers.

If using the ArcGIS Enterprise administrator APIs for creating custom data feature services, you will need to ensure that the service.json file includes these parameters in the jsonProperties.customDataProviderInfo.serviceParameters object as shown below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

...rest of service.json

"jsonProperties": {"customDataProviderInfo": {
  "forwardUserIdentity": false,
  "dataProviderName": "my-provider",
  "serviceParameters": {
   "doc_id": "parameter value",
   "geom_value": "parameter value",
   "coordinate_system": "parameter value"
   }

 }},

... rest of service.json

The values for these parameters are accessible in the provider's model.js methods just as the former host and id were on the req.params object. Following the example above, the Google Sheets document id could be accessed in the getData(req) method by req.params.doc_id.

Development and Testing

When developing and testing the provider locally with the Custom Data Feeds CLI tool, you will need to use a client that can send the appropriate headers to simulate what happens in a production environment. Include a header named x-esri-cdf-service-params whose value is an object with key and value pairs. For example: {"doc_id":"1234ABCD","coordinate_system":"4326"}.

The authorize() Method and the _user Object

The authorize() method is an optional method that can be employed in the model.js file. If used, authorize() will be executed before getData(). Inside this method, throwing an error will end the request, and neither the getData() or the editData() methods will be executed.

Here is an example of how this method could be used to only allow requests to proceed for a specific set of allowed users.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

const cdConfig = require('../cdconfig.json');
class Model {
  #allowedUsers = [
    'johnDoe1',
    'johnDoe2',
    'johnDoe3'
  ]
  constructor({logger}) {
    this.logger = logger;
  }

  async authorize(req) {
    const requestUsername = req._user?.username;

    if(this.#allowedUsers.includes(requestUsername)) {
      this.logger.info(`In Provider "${cdConfig.name}", "authorize" method: pass`)
      return;
    }

    this.logger.info(`In Provider "${cdConfig.name}", "authorize" method: reject`)
    const error = new Error('Unauthorized');
    error.code = 403;
    throw error;

  }

  ...

The _user object is only available on the request object if the property forwardUserIdentity is set to true in the service JSON. This property must be set by an administrator and is not set by default. Set the property in the jsonProperties.customDataProviderInfo object as seen below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

...rest of service.json

"jsonProperties": {"customDataProviderInfo": {
  "forwardUserIdentity": true,
  "dataProviderName": "my-provider",
  "serviceParameters": {
   "doc_id": "parameter value",
   "geom_value": "parameter value",
   "coordinate_system": "parameter value"
   }

 }},

... rest of service.json

After setting forwardUserIdentity to true, the _user object will be availabe on the request object in the getData() and editData() methods as well. User information that is available depends on whether the intial request was authorized (request made with a valid token) or unauthorized (request made without a valid ArcGIS Enterprise Server or Portal token). See the table below for expected properties available on the _user object.

Development and Testing

When developing and testing the provider locally with the Custom Data Feeds CLI tool, you will need to use a client that can send the appropriate headers to simulate what happens in a production environment. Include a header named x-esri-request-user whose value is an object with key and value pairs. For example: {"username":"johndoe1","role":"user","id":"ABDC1234"}.

Supporting Multiple Layers

Custom data providers can be written such that feature services that use them can support multiple layers. Refer to the simplified example below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
class Model {
  #logger;

  constructor({ logger }) {
    this.#logger = logger;
  }

  async getData(req) {
    const layer = req.params.layer;
    let response;

    switch (layer) {
      case '0':
        response = this.getLayer0Data();
        break;
      case '1':
        response = this.getLayer1Data();
        break;
      default:
        response = this.getServiceLevelData();
        break;
    }

    return response;
  }

  async getLayer0Data() {
    // code for retrieving and transforming layer 0 data
    return {
      ...geojson,
      metadata: metadata
    };
  };

  getLayer1Data() {
    // code for retrieving and transforming layer 1 data
    return {
      ...geojson,
      metadata: metadata
    };
  }

  getServiceLevelData() {
    return {
      layers: [this.getLayer0Data(), this.getLayer1Data()],
      tables: [],
      metadata: {
        inputCrs: 4326,
        name: "Simple multiple layer service.",
        description: "This cdf service uses two layers."
      }
    }
  }
}

module.exports = Model;

This example above shows the two major steps needed to implement multiple layers:

1. Logic for identifying which layer the request is being sent to.

   The layer index can be retrieved from the request object using req.params.layer. Code has to be written specifically to handle requests to each layer that the service supports. In the example above, a switch statement is used to call the function that fetches and formats data for the particular layer.

2. Returning layer information and metadata for service level request.

   If a request is made to the feature service, the response must contain the information for all the layers in the service, any tables, and metadata for the features service. The response object must contain a layers array, a tables array, and a metadata object as shown above.