Data sources

Data source defines how your widget accesses data. For example, when data is from a remote server, the widget can use a data source class to query the data to the client side. When data is generated by a widget, it can be used by another widget by putting itself into a data source class.

At a high level, a data source has a schema, some records, and may have some child/parent data source. In addition, every data source has a type, id, and status to help identify it. The DataSource interface is defined in the jimu-core package; it defines some of the following methods and properties:

• id: the data source id.
• type: property used in the widget to check which type of data source it is using.
• fetchSchema: every data source must implement this method to return the schema. This is the schema defined in the remote data service. For example, when a user adds a data source in ArcGIS Experience Builder, the data source schema is not saved in the app config; instead, the fetchSchema method is called to get the latest data source schema.
• getSchema: method used by the widget to access the data source schema and fields in the data source.
• getRecords: method used by the widget to access the data records in the data source.
• getStatus: the widget will use this method to get the data status of the data source. The statuses include NotReady, Unloaded, Loading, Loaded, and LoadError. The NotReady is for the widget output data source only. When an output data source instance is created, the instance is not ready for use. The widget that generates the output data source should dispatch an action (DataSourceStatusChanged) to make the data source ready for use. If a data source status is not ready, the query should return an empty result. For other queriable data sources, the default status is Unloaded. It will be changed to Loading when data is requested, and will be changed to Loaded or LoadError finally.

Data source is managed by the DataSourceManager to create and get data sources.

To simplify the use of a data source, a DataSourceComponent component is defined. This component accepts the useDataSource property and returns the data source object and its status info through a callback. It also accepts a function as its child, which can be used to get the data source object and info to render the data in the data source. The DataSourceComponent component can also accept an optional query property and will reload the data when the query is changed.

Most commonly used data services support query data, so the API defines the QueriableDataSource interface. This interface has properties including url, load and query etc. The difference between the load and query methods is that load updates the data and the status of the data source, while query queries and returns the records only.

Some ArcGIS services support query data and they have similar query behavior, so the ArcGISQueriableDataSource interface is defined to support the common operations of these services. And some sub interfaces are defined to support the specific operations of these services, such as FeatureLayerDataSource, SceneLayerDataSource, etc.

More specifically, we define the FeatureLayerDataSource data sources in Experience Builder to access a feature layer. To get the ArcGIS Maps SDK for JavaScript FeatureLayer object through the feature layer data source, you can call the createJSAPILayerByDataSource() method. The actual data of a FeatureLayerDataSource can be from a remote database, or from a collection of client side features. Both types support query.

In general, a data source is managed in two places: the data source object is managed in DataSourceManager, and the data source info is managed in a redux app store. When using DataSourceComponent, the component will call DataSourceManager to create the data source on demand, and return the data source object and dataSourceInfo by using a callback prop. In dataSourceInfo, the data source's instanceStatus, status, selectedIds, etc. can be returned.

Most of the ArcGIS server services are mapped to data source for easy access, such as MapServiceDataSource, FeatureServiceDataSource, etc. WebMap and WebScene in the ArcGIS Maps SDK for JavaScript are wrapped as WebMapDataSource and WebSceneDataSource in the jimu-arcgis package.

Data Source Set

A data source can have child data sources. As a result, you may include more than one data source to make it easier to use. This kind of data source is called a DataSourceSet. WebMapDataSource is a DataSourceSet. DataSourceSet is also a type of data source but the isDataSourceSet property is set to true. You can use getChildDataSources to get the child data sources from a parent data source, and use parentDataSource to get the parent data source from a child data source. Child data sources are created on demand, and you can use createDataSourceById(dsId) to create a child data source by its ID.

Data View

When multiple widgets connect to a single data source, a widget may want to see a local view of the data source. In this scenario, you would use a data view. The relationship between a data view and data source is very similar to the relationship between a view and a table in a relational database. Although a data source in Experience Builder is a view of the actual remote data source on the client-side, you can think of a data source as a table and data view as a view. An app creator can create a data view from a data source and connect a widget to the data view in the builder. At the API level, the data view is managed by using the DataSource class, so the data view has the same interface and behavior with the data source just with some properties differences. The data source that the data views based upon is called main data source, you can use getMainDataSource to get it from a data view, or use getDataViews to get all of the views of the main data source.

The selection is shared between the main data source and all of its data views. The selection info is saved in two places: the selected records are saved in a selection view of the data source, the selected record ids are saved in the redux app store, under its data source id. The selection view is a special data view of the main data source, it's identified by ${mainDataSourceId}-selection.

Local data source and data view

Although multiple widgets can use different data views to retrieve different data, there are still scenarios that different widgets may need to connect to the same data source or the same data view but they still need to retrieve different data in the widget, such as a drop-down list in a widget. In this case, the widget can create a local data source or data view, the filter applied on the local data source or data view does not affect the related data source or data view. You can use DataSourceComponent and pass in localId, or use DataSourceManager().getinstance().createLocalDataSource to use the local data source or data view.

Source records in data source

In most cases, the data is in a remote database, the data source instance just retrieves the data from the remote and stores it in the data source instance. However, for some data sources, the data is generated on the client-side such as a widget's output data sources and the selection view data source. In these scenarios, the data of the data source is stored in the sourceRecords of the data source instance. You can use getSourceRecords and setSourceRecords to get and update it.

Widget output data source

A widget can use a data source, and it can generate a data source as well, we call the generated data source a widget output data source. A widget must save its output data source to the app config in its setting page by calling this.props.onSettingChange so other widgets can use its output data source. The output data source is saved in dataSources in the app config. Other widgets should not see any differences when they use the output data source or use the user added data source.

Internally, when a widget generates an output data source and when the widget is rendered, an output data source and an output data view are created. The output data view is identified by ${outputDataSourceId}-output. The widget that uses the output data source is using the output view. When an output data source is created, its status is NotReady. The widget that generates the output data source should update the data source when the data is ready, and then change the data source status to Unloaded so other widgets know it's ready to use. The way to update the data in the output data source depends on how the output data source is generated. A widget can generate a server-side output data source, which connects to a remote service. In this case, you can use outputDs.updateQueryParams(queryParams, widgetId), outputDs.load(queryParams, { widgetId }) or <DataSourceComponent useDataSource={Immutable({ dataSourceId: outputDsId, mainDataSourceId: outputDsId })} query={queryParams}> to update the data source. A widget can generate a client-side output data source, which stores the data in the client-side. In this case, you can use outputDs.setSourceFeatures(features, options) or outputDs.setSourceRecords(records) to update data, depending on what kind of data the widget has generated. If the data generated by the widget has geometry, the setSourceFeatures is recommended because you can specify the renderer for the features in this method. If the data generated by the widget does not have geometry, both ways are fine.

The originDataSources in the output data source JSON is used to maintain the relationship between the origin data source and the output data source. For example, if a widget's output data source does not have a schema defined, the origin data source's schema will be used. The widget should update this property and the general properties including the id, type, etc. The Query and Chart widget both generate an output data source. You can use their output data source JSON as a reference to see which properties are required in the output data source JSON.

Query and filter data

When a widget needs to load data, the recommended way is to use the <DataSourceComponent> component or dataSource.load() because the data source framework helps you manage the data cache and pagination issues. If you just want to filter data instead of loading data, you can use dataSource.updateQueryParams().

Query fields

A feature layer may have many fields, but in most cases widgets only need data from a few of those fields. In most cases, a widget may need to query and display many data records. The recommended way for this kind of widget is to save the fields configurated in the settings to its useDataSources via the onSettingChange method, and then use the <DataSourceComponent> to query and render the data. When taken this way, only the saved fields will be queried. When multiple widgets use the same data source, all saved fields will be queried together. However, some widgets may need to just display one record and lots of fields. In this case, the recommended way is to use the dataSource.query() with the record ID to query the record to avoid querying too many fields when multiple widgets connect to the same data source. If you use dataSource.query(queryParam) to query data, you can set the outFields property in the queryParam object to specify the fields you want to query.

Client-side query

When a data source created from an URL needs to query data, the data source will send the query to the remote server. However, when the data source is created from a map layer and the Map widget has enabled client-side query, the data source will try to query data from the MapView instead of sending a query to the back-end service. If the MapView or the corresponding LayerView is not ready for a client-side query, the data source will still send the query to the remote server. There are couple of reasons why the MapView or the LayerView may not be ready. For example, the geometry in the query is not in the current map extent, or the LayerView does not have all the features. Before performing the client-side query, the LayerView will add the outFields in the query params to the LayerView's fields and then perform the query. Adding too many fields to LayerView may cause performance issues, and you can use notAddFieldsToClient to prevent adding fields to the LayerView.

Data Action

See Data action

Repeated Data source

The data source provided by RepeatedDataSourceProvider is called a repeated data source. All children widgets of the widget that provide a data source will receive the repeated data source. This is similar to React's Context. A widget can access repeated data source by this.props.repeatedDataSource. The repeated data source will get the data source's id, record, and recordIndex. Any widget can provide repeated data source by using RepeatedDataSourceProvider. The List widget in Experience Builder is a good example that provides a repeated data source. To use a repeated data source, add the supportRepeat property in a widget's manifest file.