Data sources

What is a data source?

A data source in ArcGIS Experience Builder defines how your widget accesses and manages data. Data sources provide a standardized way for widgets to interact with data types, whether from remote servers, local collections, or generated by other widgets.

Data sources serve as the bridge between your widgets and data, handling:

• Remote data access: Querying data from external services and databases.
• Local data management: Storing and managing client-side data collections.
• Widget communication: Sharing data between widgets through output data sources.

Data source structure

Every data source has a consistent structure with these components:

• Schema: Defines the structure and fields of the data.
• Records: contains the data queried from the source according to the current query criteria, can use the getRecords() to get it.
• Status: Indicates the data source instance status and the data loading status. For the instance status, it includes the NotCreated, Created, and CreateError. For the data loading status, it includes NotReady, Unloaded, Loading, Loaded, LoadError
• Type and ID: Identifies the data source type and unique identifier.

The DataSource interface, defined in the jimu-core package, provides these methods:

• id: The unique identifier for the data source.
• type: Property used to identify the data source type.
• getSchema(): Accesses the data source schema and fields.
• getRecords(): Retrieves data records from the data source.
• getStatus(): Returns the current status (NotReady, Unloaded, Loading, Loaded, LoadError).

Data Source Management: Data sources are managed by the DataSourceManager to create and get data sources. The data source object is managed in DataSourceManager, while the data source info is managed in a redux app store. When using DataSourceComponent, the component calls DataSourceManager to create the data source on demand and returns the data source object and dataSourceInfo through a callback prop. The data source configuration is saved in the appConfig.dataSources, and the data source instances are managed by the data source manager.

DataSourceComponent: This component simplifies data source usage by accepting a useDataSource property and returning the data source object and its status info through a callback. It also accepts a function as its child, which can be used to get the data source object and info to render the data. The component can accept an optional query property and will reload the data when the query changes.

Data source types

Experience Builder provides many types of data sources to handle different data scenarios, below are some important data source types:

• QueriableDataSource: For data services that support querying operations
• ArcGISQueriableDataSource: For ArcGIS services with common query behaviors
• FeatureLayerDataSource: For accessing feature layer data
• MapServiceDataSource: For map service data access
• FeatureServiceDataSource: For feature service data access
• WebMapDataSource: Wraps WebMap from ArcGIS Maps SDK for JavaScript
• WebSceneDataSource: Wraps WebScene from ArcGIS Maps SDK for JavaScript

How to use data sources

Here are the steps to work with data sources in your widgets:

1. Select a data source

To select a data source for your widget:

1. Use the DataSourceSelector component in your widget settings
2. Choose from available data sources or add new ones
3. Configure the data source properties as needed

2. Configure data source

Configure how your widget uses the data source:

1. Setting query parameters to filter the data
2. Specifying which fields to retrieve
3. Configuring pagination and caching options

3. Access data

Access data in your widget using:

1. DataSourceComponent: Recommended approach for most scenarios
2. Direct methods: Use dataSource.load() or dataSource.query() for specific needs
3. Status monitoring: Check data source status before accessing data

Advanced data source concepts

Advanced data source concepts provide patterns for composing, scoping, and sharing data so widgets can work efficiently and independently. They include data source sets, data views, local data sources, source records, and widget output data sources. Despite their differences, each still conforms to the DataSource interface and is created, tracked, and updated through the DataSourceManager, giving you a consistent API for querying, loading, filtering, and status management.

Data source sets

A DataSourceSet is a data source that contains child data sources, making it easier to work with related data collections. WebMapDataSource is a DataSourceSet.

Key features of data source sets:

• Contains multiple related data sources
• Child data sources are created on demand
• Use getChildDataSources to retrieve child data sources
• Use parentDataSource to access the parent data source
• Create child data sources with createDataSourceById(dsId)

Data views

Data views provide local views of data sources, similar to views in relational databases. They allow multiple widgets to work with the same data source while maintaining different perspectives.

Data view characteristics:

• Created from a main data source
• Share the same interface as data sources
• Selection is shared between main data source and all views
• Use getMainDataSource to access the main data source

Local data sources

Local data sources and data views allow widgets to work with filtered data without affecting the original data source. This is useful for scenarios like dropdown lists that need different data views.

To use local data sources:

• Use DataSourceComponent with a localId parameter
• Use DataSourceManager().getinstance().createLocalDataSource()
• Filters applied to local data sources do not affect the original data source

Widget output data sources

Widgets can generate output data sources that other widgets can consume. This enables data flow between widgets in your application.

To create output data sources:

1. Save the output data source configuration in your widget settings using this.props.onSettingChange
2. The output data source is saved in the dataSources section of the app config
3. Other widgets can use the output data source like any other data source

Important: When an output data source is created, its status is NotReady. The widget that generates the output data source should update the data source when the data is ready, and then change the data source status to Unloaded so other widgets know it is ready to use.

Output data source types:

• Server-side: Connects to remote services

  • Use outputDs.updateQueryParams(queryParams, widgetId)
  • Use outputDs.load(queryParams, { widgetId })
  • Use <DataSourceComponent useDataSource={Immutable({ dataSourceId: outputDsId, mainDataSourceId: outputDsId })} query={queryParams}>

• Client-side: Stores data locally

  • Use outputDs.setSourceFeatures(features, options) to set the source if you have some features
  • Use outputDs.setSourceRecords(records) to set the source if you have some data source records

The originDataSources property in the output data source JSON maintains the relationship between the origin data source and the output data source. If an output data source for a widget does not have a schema defined, the schema of the origin data source will be used.

Data querying and filtering

Loading and filtering data

When working with data sources, you have options for data access:

• Loading data: Use DataSourceComponent or dataSource.load() for data loading with caching and pagination. The load method updates the data and status of the data source.
• Filtering data: Use dataSource.updateQueryParams() to apply filters without reloading
• Querying data: Use dataSource.query() for queries that return records. The query method queries and returns records without updating the data source status.

Field selection

Optimize performance by selecting only the fields you need:

• Multiple records, few fields: Save field configuration in the widgetJson.useDataSources[i].fields in settings
• Single record, many fields: Use dataSource.query() with record ID
• Custom field selection: Set outFields property in query parameters

Client-side querying

When working with map layers, data sources can perform client-side queries for performance:

• Queries are executed against the MapView instead of remote servers
• Falls back to server queries if MapView or LayerView is not ready
• Use notAddFieldsToClient to prevent performance issues with many fields
• Before performing client-side queries, the LayerView adds the outFields from query params to the LayerView fields

Code examples

Here are some code examples for working with data sources:

1
2
3
4
5
6
7
8
9
10
11
12
<DataSourceSelector
  mustUseDataSource
  types={Immutable([DataSourceTypes.FeatureLayer])}
  onChange={(useDss: UseDataSource[]) => {
    props.onSettingChange({
      id: props.id,
      useDataSources: useDss
    })
  }}
  useDataSources={props.useDataSources}
  widgetId={props.id}
/>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
<DataSourceComponent
  useDataSource={props.useDataSources[0]}
  widgetId={props.id}
  query={customQueryParameters}
>
{
  (ds, info: IMDataSourceInfo) => {
    return <div>
      <div>Query state: {info.status}</div>
      <div className='record-list'>
        {
          ds
            ? ds.getRecords().map((r, i) => {
              return <div key={i}>{r.getData()[props.config.fieldName]}</div>
            })
            : null
        }
      </div>
    </div>
  }
}
</DataSourceComponent>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
onChange = () => {
  const outputDsJsons: DataSourceJson[] = [{
    id: `${props.id}-ouput`,
    type: DataSourceTypes.FeatureLayer,
    sourceLabel: `${props.manifest.name}-output-data-source`,
    isOutputFromWidget: true,
    geometryType: 'esriGeometryPoint',
    originDataSources: [useDataSources[0]], // Optional, set it if creating this output data source depends on other data sources.
    schema: schema, // Optional, set it if the output data source is client-side (without url).
    url: 'https://<root>/<serviceName>/FeatureServer/<layerId>', // Optional
    layerId: '<layerId>', // Optional
    itemId: itemId, // Optional
    portalUrl: portalUrl // Optional
  }]

  props.onSettingChange({
    id: props.id
  }, outputDsJsons)
}

1
2
3
4
5
6
7
8
9
onFeaturesChange = (features: IFeature[]) => {
  const outputDs = DataSourceManager.getInstance().getDataSource(this.props.outputDataSources?.[0])
  if (!outputDs) return
  // If the output data source is client-side (without url), you can update its records by this way.
  outputDs.setSourceRecords(features.map(f => outputDs.buildRecord(f)))
  // Status of output data source is NotReady by default, set it to Unloaded to let other widgets know output data source is ready to do query.
  outputDs.setStatus(DataSourceStatus.Unloaded)
  outputDs.setCountStatus(DataSourceStatus.Unloaded)
}

Data actions

Data actions allow widgets to perform operations on data sources. For information about implementing data actions, see Data action.

Repeated data sources

Repeated data sources work similarly to React Context, allowing parent widgets to provide data to child widgets.

To use repeated data sources:

• Access via this.props.repeatedDataSource
• Contains id, record, and recordIndex properties
• Use RepeatedDataSourceProvider to provide repeated data sources
• Add supportRepeat property to your widget manifest file

The List widget is an example of a widget that provides repeated data sources.