Create a MongoDB Custom Data Feed with Query and Editing Capabilities

In this tutorial, you will create a custom data provider that allows for creating, reading, updating, and deleting point feature data hosted in a MongoDB instance. The source code used in this tutorial can be accessed here.

Prerequisites

• Install ArcGIS Server and the ArcGIS Server Custom Data Feeds Runtime on the same machine.
• Install a version of Node.js between 16.19.1 and 20.17.0 on your development machine depending on your ArcGIS version. See Installing and configuring for version details.
• Install the ArcGIS Enterprise SDK on your development machine. It is highly recommend that your development machine and server machine are running the same operating system and the same version of Node.js.
• Setup your own MongoDB instance and populate it with the provided dataset. The data are found in data/ wildfires-sample.json. This tutorial assumes a local deployment of MongoDB. Edit your connectString accordingly below in the configuration.
• It is recommended that you complete the Yelp Walkthrough before this tutorial in order to get a foundational understanding of how to create a custom data provider.

How the Provider Works

This sample provider will read and write to a MongoDB instance that is populated with documents in the format below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
  "_id": "1150d48a-4283-409f-95dc-d97487b8ad84",
  "fireId": "AK6448214467319840607",
  "fireName": "GLACIER CREEK",
  "fireType": "Wildfire",
  "acres": 1897,
  "location": {
    "type": "Point",
    "coordinates": [
      -144.67327160022626,
      64.48231108796996
    ]
  },
  "alternateID": 212345
}

Notice in the document above the attribute alternateID. ArcGIS does not currently support strings as ObjectIDs, so the _id field generated by MongoDB cannot be used. If you inspect the sample code, there is a function that hashes the _id and saves the hashed integer as the alternateID in the document. This alternateID is used as the ObjectID. See the API Reference for details on designating a feature attribute as the ObjectID with the idField metadata property.

Create a Custom Data App

Complete the following steps to create a custom data app named mongodb-app.

1. On your development machine, open a command prompt and navigate to a directory where you want your project to be saved.

2. Create the app by running the command: cdf createapp mongodb-app.

   

3. Verify your app runs by running the command: npm start.

   

4. Once you verify that the app is running, stop it by running the command: ctrl + c.

Create a Custom Data Provider

1. After you've created a custom data app, open a command prompt in the app directory and run the cdf createprovider editable-mongodb-points command. This will create an editable-mongodb-points directory that includes a project template with boilerplate code.

   

Install required packages

1. In the command prompt, navigate to the providers/editable-mongodb-points directory, and run the command npm install config mongodb proj4 @esri/proj-codes to install the needed modules. proj4 and @esri/proj-codes are tools that are used for coordinate system conversion.

   

Configure the Provider

1. In the providers/editable-mongodb-points/cdconfig.json file, set the value of the properties.hosts field to true and properties.disableIdParam field to false.

2. In the providers/editable-mongodb-points/config directory, configure your MongoDB connection in the default.json file. It will look similar to this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
    "editable_mongodb_points": {
        "connectString": "mongodb://127.0.0.1:27017",
        "definedCollectionsOnly": true,
        "databases": {
            "editable-sample-fires": {
                "fires": {
                    "geometryField": "location",
                    "idField": "alternateID",
                    "cacheTtl": 0,
                    "crs": 4326,
                    "maxRecordCount": 2000
                }
            }
        }
    }
}

Implement the Provider Code

Click here to acces the source code and dataset for this tutorial.

Enable the applyEdits Routes

For detailed documentation on how the editing capability is achieved in custom data providers, click here.

1. Create a new file called routes.js in the providers/editable-mongodb-points/src directory. Add the following code to the file:

   Use dark colors for code blocksCopy

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   module.exports = [
       {
           path: `/editable-mongodb-points/rest/services/:host/:id/FeatureServer/:layer/applyEdits`,
           methods: ['POST'],
           handler: 'editData'
       },
       {
           path: `/editable-mongodb-points/rest/services/:host/:id/FeatureServer/applyEdits`,
           methods: ['POST'],
           handler: 'editData'
       }
   ]

2. Create a new file called controllers.js in the providers/editable-mongodb-points/src directory. Add the following code to the file:

   Use dark colors for code blocksCopy

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   function Controller (model) {
   this.model = model
   }
   
   Controller.prototype.editData = async function (req, res) {
       const pathParams = req.params;
       const body = req.body;
       const reply = await this.model.editData(pathParams, body);
       res.status(200).json(reply);
   }
   
   module.exports = Controller

3. Open the file index.js in the providers/editable-mongodb-points/src directory. Add the two lines (routes and Controller) shown in the code below to the provider object.

   Use dark colors for code blocksCopy

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   const packageInfo = require('../package.json')
   const csconfigInfo = require('../cdconfig.json')
   
   const provider = {
     type: csconfigInfo.type,
     name: csconfigInfo.name,
     version: packageInfo.version,
     hosts: csconfigInfo.properties.hosts,
     disableIdParam: csconfigInfo.properties.disableIdParam,
     Model: require('./model'),
     routes: require('./routes'),
     Controller: require('./controllers')
   }
   
   module.exports = provider

Implement the model.js file

Replace the contents of the default providers/editable-mongodb-points/src/model.js with the provided sample code. Due to the size of the files in the src directory, they are not shown in their entirety on this page, but important aspects of the code are highlighted in this tutorial.

The editData function recieves that path parameters to specify which database and collection to connect to, coordinates the helper functions, and returns the response.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

  // *code snippet*

  async editData(pathParams, body) {

    // assign database and collection name from path parameters
    const databaseName = pathParams.host;
    const collectionName = pathParams.id;

    // add logic to normalize layer-level or service-level requests
    const extractedEdits = normalizeRequestedEdits(body);

    const database  = this.#client.db(databaseName);
    const collection = database.collection(collectionName);

    let applyEditsResponse = {};  // initialize the response object

    // call the necessary functions to handled the edits
    applyEditsResponse = await performEdits(collection, extractedEdits.edits);

    // check if the response should be an object(layer-level) or an array(service-level)
    if (extractedEdits.editLevel === 'service') {
      applyEditsResponse.id = extractedEdits.layer;
      return [applyEditsResponse];

    }

    return applyEditsResponse;

  }

  // *code snippet*

The getData function is a simple, full-fetch implementation that retrieves all the data from the table. The geoJSON metadata contains the idField property that points to the attribute that is used at the ObjectID, a templates array with the object that defines what the editing widget will look like, and the fields array that specifies the properties of each attribute.

The empty object for the value of prototype.attributes in the templates metadata specifies that all attributes will be shown in the editing widget.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

    // *code snippet*

        return {...geojson, metadata: {
            idField: 'alternateID',
            name: 'Fires',
            templates: [
                {
                    "name": "Edit MongoDB Fires",
                    "description": "Template for editing fire data features",
                    "drawingTool": "esriFeatureEditToolPoint",
                    "prototype": {
                        "attributes": {}
                    }
                }
            ],

    // *code snippet*

All feature attributes must be listed in the fields array with correct type and marked editable if desired. Notice that the alternateID is larger than a standard integer, and therefore specified as a bigInteger.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
    // *code snippet*

        fields: [
          {
            "name": "_id",
            "type": "string",
            "alias": "mongoID",
            "length": 128,
            "editable": false
          },
          {
            "name": "alternateID",
            "type": "bigInteger",
            "alias": "alternateID",
            "editable": false
          },
          {
              "name": "fireId",
              "type": "string",
              "alias": "fireId",
              "length": 128,
              "editable": true
          },
          {
              "name": "fireName",
              "type": "string",
              "alias": "fireName",
              "length": 128,
              "editable": true
          },
          {
              "name": "fireType",
              "type": "string",
              "alias": "fireType",
              "length": 128,
              "editable": true
          },
          {
              "name": "acres",
              "type": "string",
              "alias": "acres",
              "length": 128,
              "editable": true
          }
        ]

    // *code snippet*

Implement the classes and constants files

Copy the contents of response-classes.js to providers/editable-mongodb-points/src/classes. This file contains various classes for success and error responses.

Copy the contents of constants.js to providers/editable-mongodb-points/src/constants. This file contains the wkid for the data in the Mongo database.

Implement the helper files

Copy the contents of the seven files: document-delete-helpers.js, document-insert-helpers.js, document-update-helpers.js, getDataHelpers.js, index.js, normalize-edits-request.js, perform-edits.js to to providers/editable-mongodb-points/src/helpers.

Several ancillary operations are carried out in these helper files including maintaining spatial data integrity and maintaining document object shape. See in the snippet below that the spatial reference of the incoming edit from the client (record) is checked against that of the remote data by checking the wkid of each and performing a conversion if necessary.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

    // *code snippet*

    const codes = require('@esri/proj-codes');
    const proj4 = require('proj4');

    // other code not shown

    function transformToDatasourceJson(record) {

        // check if the incoming SR is the same as the data SR; if differnt, convert and modify the record
        const featureSR = record.geometry.spatialReference.wkid;
        if (featureSR !== CONSTANTS.SOURCE_CRS_WKID) {

            // look up the code
            const crs = codes.lookup(featureSR);
            // convert coordinates from what is currently in client to our data source crs
            const convertedCoordinates = proj4(crs.wkt,`EPSG:${CONSTANTS.SOURCE_CRS_WKID}`, [record.geometry.x, record.geometry.y]);
            // push the converted coordinates into the array
            record.geometry.x = convertedCoordinates[0];
            record.geometry.y = convertedCoordinates[1];

        }

        const transformedJson = new DataBaseObject(
            record.attributes.fireId.toString(),
            record.attributes.fireName.toString(),
            record.attributes.fireType,
            record.attributes.acres,
            record.geometry.x,
            record.geometry.y
        );

        return transformedJson;

    }

    // *code snippet*

Test the Provider

After implementing all the steps above, the providers/editable-mongodb-points/ directory should have the following structure:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
    - config/
        | default.json
    - node_modules/
    - src/
        - classes/
            | response-classes.js
        - constants/
            | constants.js
        - helpers/
            | document-delete-helpers.js
            | document-insert-helpers.js
            | document-update-helpers.js
            | getDataHelpers.js
            | index.js
            | normalize-edits-request.js
            | perform-edits.js
        | controllers.js
        | index.js
        | model.js
        | routes.js
    | cdconfig.json
    | package-lock.json
    | package.json

1. Navigate to the mongodb-app directory in a command prompt, and run the npm start command to start the custom data app.
2. In a web browser, navigate to http://localhost:8080/editable-mongodb-points/rest/services/sample-data/fires/FeatureServer/0/query, and verify that the MongoDB provider is returning data points.

Build and Deploy the Custom Data Provider Package File

3. Stop the custom data app if it is running.
4. Open a command prompt and navigate to the custom data app directory.
5. Run the cdf export editable-mongodb-points command.
6. In a web browser, navigate to the ArcGIS Server Administrator Directory and sign in as an administrator.
7. Click uploads > upload.
8. On the Upload Item page, click Choose File and select the editable-mongodb-points.cdpk file. Optionally, provide a description in the Description text box.
9. Click Upload. Once the file is uploaded, you will be directed to a page with the following header: Uploaded item - <item_id> . Copy the item id.
10. Browse back to the root of the Administrator Directory and then click services > types > customdataproviders.
11. On the Registered Customdata Providers page, click register and paste the item id into the Id of uploaded item field.
12. Click Register.

Create Feature Service

1. Browse back to the root of the Administrator Directory and click services > createService.

2. On the Create Service page, copy and paste the following JSON into the Service (in JSON format) text box.

   Use dark colors for code blocksCopy

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   16
   17
   18
   19
   20
   21
   22
   23
   24
   25
   26
   27
   28
   29
   30
   31
   32
   33
   34
   35
   36
   37
   38
   {
       "serviceName": "mongoEditPoints",
       "type": "FeatureServer",
       "description": "",
       "capabilities": "Query",
       "provider": "CUSTOMDATA",
       "clusterName": "default",
       "minInstancesPerNode": 0,
       "maxInstancesPerNode": 0,
       "instancesPerContainer": 1,
       "maxWaitTime": 60,
       "maxStartupTime": 300,
       "maxIdleTime": 1800,
       "maxUsageTime": 600,
       "loadBalancing": "ROUND_ROBIN",
       "isolationLevel": "HIGH",
       "configuredState": "STARTED",
       "recycleInterval": 24,
       "recycleStartTime": "00:00",
       "keepAliveInterval": 1800,
       "private": false,
       "isDefault": false,
       "maxUploadFileSize": 0,
       "allowedUploadFileTypes": "",
       "properties": {
           "disableCaching": "true"
       },
       "jsonProperties": {
           "customDataProviderInfo": {
               "dataProviderName": "editable-mongodb-points",
               "dataProviderHost": "editable-sample-fires",
               "dataProviderId": "fires"
           }
       },
       "extensions": [],
       "frameworkProperties": {},
       "datasets": []
   }

3. Click Create.

4. In ArcGIS Server Administrator Directory, navigate to Home > services > mongoEditPoints > edit

5. Change the value of capabilities to Query,Editing, and click the Save Edits button.

Keep in mind that the provider code we used above assumes a database named editable-sample-fires and a collection named fires. If you used different names in your MongoDB instance, update the values of dataProviderHost and dataProviderId accordingly.

Consume Feature Service

To access the MongoDB feature service that you created in the previous section, use the appropriate URL (e.g., https://<domain_or_machine_name>/<webadaptor_name>/rest/services/mongoEditPoints/FeatureServer). You can use this URL to consume data from MongoDB in ArcGIS clients like ArcGIS Pro, ArcGIS Online, and ArcGIS Enterprise.

To explore the editing capabilities of the provider in this walkthrough in a graphical user interface, add this feature service to a map in ArcGIS Map Viewer.

1. Click the editing widget and select a point on the map.

2. Notice that the attributes fireId, fireName, fireType, and acres are all editable. mongoID was not specified as editable in our geojson.medata.fields array.

3. Edit one of these attributes and click Update to change the value in the MongoDB collection.

To explore the applyEdits REST API calls in more detail, navigate the ArcGIS REST Services Directory and navigate to applyEdits at: https://<domain_or_machine_name>/<webadaptor_name>/rest/services/mongoEditPoints/FeatureServer/applyEdits

1. Enter the following in the Edits: texbox:

   Use dark colors for code blocksCopy

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   16
   17
   18
   19
   20
   21
   22
   23
   24
   25
   26
   27
   28
    [
        {
            "id": 0,
            "adds": [
                {
                    "geometry": {
                        "spatialReference": {
                            "latestWkid": 3857,
                            "wkid": 102100
                        },
                        "x": -12633717.653726798,
                        "y": 7084687.088394526
                    },
                    "attributes": {
                        "_id": null,
                        "fireId": "C3PO",
                        "fireName": "Golden Fire",
                        "fireType": "Wild",
                        "acres": "4682"
                    }
                }
            ],
            "updates": null,
            "deletes": null,
            "attachments": null,
            "assetMaps": null
        }
    ]

2. Click Apply Edits to create the new feature.

   

You've successfully created a MongoDB custom data feed that can both read from and write to a MongoDB collection.