Make your data accessible to the GeoAnalytics Server

Collecting, storing, managing, and analyzing large quantities of numbers, figures, and files is not a new business activity. However, referring to these numbers, figures, and files as big data is relatively recent.

The GeoAnalytics Server expands your ArcGIS Enterprise deployment, providing functionality and services to process and analyze big data.

In order to run the GeoAnalytics tools, your data needs to be in one of the following formats:

• Feature layers (hosted, hosted feature layer views, and from feature services)
• Feature collections
• Big data file shares registered with ArcGIS GeoAnalytics Server

Note: Please refer to the features module guide to understand more about feature layers and feature collections.

Big data file shares

The GeoAnalytics server allows you to register datasets in a format called a big data file share. Big data file shares are items on your Web GIS, and can reference data in any of the following data sources:

• File Share - a directory of datasets stored locally or shared across a network
• HDFS - a Hadoop Distributed File System directory of datasets
• Apache Hive - a metastore database
• Cloud Store - an Azure Blob Storage container or Amazon Web Services S3 bucket

Storing your data in a big data file share datastore benefits you because:

• The GeoAnalytics tools read your data only when they are executed, which allows you to update or add data to these locations.
• You can use partitioned data as a single dataset.
• Big data file shares are flexible in how time and geometry are defined, allowing data in multiple formats in a single dataset.

When writing results to a big data file share, you can use the following output GeoAnalytics Tools:

• File share
• HDFS
• Cloud store

The following file types are supported as datasets for input and output in big data file shares:

• Delimited files (such as .csv, .tsv, and .txt)
• Shapefiles (.shp)
• Parquet files (.gz.parquet)
• ORC files (orc.crc)

Store data where all ArcGIS Server machines can access it

To allow your ArcGIS Server sites to access the data resources you want to publish, all machines in the ArcGIS Server site must have access to the resource. For example, when you publish a map as a service, the map and all the data for the map's layers must be accessible to all ArcGIS Server machines.

You need to do the following to make your data accessible to ArcGIS Server:

Store your data in a location that all machines in your ArcGIS Server site can access.

Grant permissions to allow ArcGIS Server to access the data. If your data is stored in a folder or a database that you access using operating system authentication, you must grant the ArcGIS Server account permissions to these locations. The ArcGIS Server account is the account you used to install ArcGIS Server, not the primary site administrator specified when the ArcGIS Server site was created. If your data is stored in a database that you access using database authentication, the database user you provide when registering the database must have permissions to the data.

Register your data store with the ArcGIS Server site.

Prepare your data to be registered as big data file share

To register a file share or an HDFS, you need to format your datasets as subfolders within a single parent folder and register the parent folder. This parent folder becomes a datastore, and each subfolder becomes a dataset. For instance, to register 2 datasets representing earthquakes and hurricanes, your folder hierarchy would look like below:

|---FileShareFolder         <-- register as a datastore
   |---Earthquakes          <-- dataset 1
      |---1960              
         |---01_1960.csv
         |---02_1960.csv
      |---1961              
         |---01_1961.csv
         |---02_1961.csv
   |---Hurricanes           <-- dataset 2
      |---atlantic_hur.shp
      |---pacific_hur.shp

Learn more about preparing your big data file share datasets here.

# Connect to enterprise GIS
from arcgis.gis import GIS
import arcgis.geoanalytics
portal_gis = GIS("your_enterprise_portal")

Ensuring your GIS supports GeoAnalytics

Before executing a tool, we need to ensure an ArcGIS Enterprise GIS is set up with a licensed GeoAnalytics server. To do so, call the is_supported() method after connecting to your Enterprise portal. See the Components of ArcGIS URLs documentation for details on the urls to enter in the GIS parameters based on your particular Enterprise configuration.

# Verify that GeoAnalytics is supported 
arcgis.geoanalytics.is_supported()

True

Registering big data file shares

The get_datastores() method of the geoanalytics module returns a DatastoreManager object that lets you search for and manage the big data file share items as Python API Datastore objects on your GeoAnalytics server.

bigdata_datastore_manager = arcgis.geoanalytics.get_datastores()
bigdata_datastore_manager

<DatastoreManager for https://pythonapi.playground.esri.com/ga/admin>

The data prepared above can be stored to one of the following locations based on the size of your data, the number of people who will access the web service, and how frequently the data changes.

• Store data locally on each ArcGIS Server machine
• Store data in a shared directory
• Store data in a database
• Store caches, imagery, and big data files in a cloud storage container

Later, We will learn how to register a cloud store as datastore for registering your data as big data file share. Next, we will learn how to register data from a local/shared directory.

Register the cloud store with your GeoAnalytics Server

conn_dict = {"accessKeyId":"<provide key here>",
             "secretAccessKey":"<provide secret key here>",
             "region":"<provide region here>",
             "defaultEndpointsProtocol":"<probide https or http here>",
             "credentialType":"accesskey"}

import json
conn_str = json.dumps(conn_dict)

datastore_obj = bigdata_datastore_manager.add_cloudstore(name='cloud_store', 
                                         conn_str=conn_str, 
                                         object_store="esri-delhi-store", 
                                         provider='amazon')

Created cloud store for cloud_store

datastore_obj.path

'/cloudStores/cloud_store'

data_item1 = bigdata_datastore_manager.add_bigdata(name="ServiceCallsOrleans", 
                                                  server_path=data_item.path, 
                                                  connection_type='dataStore')

Created Big Data file share for ServiceCallsOrleans

data_item2 = bigdata_datastore_manager.add_bigdata("ServiceCallsOrleans", r"\\machinename\datastore")

Created Big Data file share for ServiceCallsOrleans

bigdata_fileshares = bigdata_datastore_manager.search()
bigdata_fileshares

[<Datastore title:"/bigDataFileShares/NYC_taxi_data15" type:"bigDataFileShare">,
 <Datastore title:"/bigDataFileShares/all_hurricanes" type:"bigDataFileShare">,
 <Datastore title:"/bigDataFileShares/NYCdata" type:"bigDataFileShare">,
 <Datastore title:"/bigDataFileShares/hurricanes_1848_1900" type:"bigDataFileShare">,
 <Datastore title:"/bigDataFileShares/ServiceCallsOrleans" type:"bigDataFileShare">,
 <Datastore title:"/bigDataFileShares/hurricanes_dask_csv" type:"bigDataFileShare">,
 <Datastore title:"/bigDataFileShares/hurricanes_dask_shp" type:"bigDataFileShare">,
 <Datastore title:"/cloudStores/cloud_store" type:"cloudStore">]

file_share_folder = bigdata_fileshares[4]
file_share_datasets = file_share_folder.datasets
len(file_share_datasets)

1

for i in range(0, len(file_share_datasets)):
    print("{:<10}{:<3}{}".format("Dataset " + str(i) + ":", "", file_share_datasets[i]['name']))

Dataset 0:   yearly_calls

# let's view the json schema of the calls dataset for a sample
file_share_datasets[0]

{'name': 'yearly_calls',
 'format': {'quoteChar': '"',
  'fieldDelimiter': ',',
  'hasHeaderRow': True,
  'encoding': 'UTF-8',
  'escapeChar': '"',
  'recordTerminator': '\n',
  'type': 'delimited',
  'extension': 'csv'},
 'schema': {'fields': [{'name': 'NOPD_Item', 'type': 'esriFieldTypeString'},
   {'name': 'Type_', 'type': 'esriFieldTypeString'},
   {'name': 'TypeText', 'type': 'esriFieldTypeString'},
   {'name': 'Priority', 'type': 'esriFieldTypeString'},
   {'name': 'MapX', 'type': 'esriFieldTypeDouble'},
   {'name': 'MapY', 'type': 'esriFieldTypeDouble'},
   {'name': 'TimeCreate', 'type': 'esriFieldTypeString'},
   {'name': 'TimeDispatch', 'type': 'esriFieldTypeString'},
   {'name': 'TimeArrive', 'type': 'esriFieldTypeString'},
   {'name': 'TimeClosed', 'type': 'esriFieldTypeString'},
   {'name': 'Disposition', 'type': 'esriFieldTypeString'},
   {'name': 'DispositionText', 'type': 'esriFieldTypeString'},
   {'name': 'BLOCK_ADDRESS', 'type': 'esriFieldTypeString'},
   {'name': 'Zip', 'type': 'esriFieldTypeBigInteger'},
   {'name': 'PoliceDistrict', 'type': 'esriFieldTypeBigInteger'},
   {'name': 'Location', 'type': 'esriFieldTypeString'}]},
 'geometry': {'geometryType': 'esriGeometryPoint',
  'spatialReference': {'wkid': 102682, 'latestWkid': 3452},
  'fields': [{'name': 'MapX', 'formats': ['x']},
   {'name': 'MapY', 'formats': ['y']}]},
 'time': {'timeType': 'instant',
  'timeReference': {'timeZone': 'UTC'},
  'fields': [{'name': 'TimeCreate', 'formats': ['MM/dd/yyyy hh:mm:ss a']}]}}

file_share_folder.datapath

'/bigDataFileShares/ServiceCallsOrleans'

file_share_folder.validate()

True

manifest = file_share_folder.manifest
manifest

{'datasets': [{'name': 'calls',
   'format': {'quoteChar': '"',
    'fieldDelimiter': ',',
    'hasHeaderRow': True,
    'encoding': 'UTF-8',
    'escapeChar': '"',
    'recordTerminator': '\n',
    'type': 'delimited',
    'extension': 'csv'},
   'schema': {'fields': [{'name': 'NOPD_Item', 'type': 'esriFieldTypeString'},
     {'name': 'Type_', 'type': 'esriFieldTypeString'},
     {'name': 'TypeText', 'type': 'esriFieldTypeString'},
     {'name': 'Priority', 'type': 'esriFieldTypeString'},
     {'name': 'MapX', 'type': 'esriFieldTypeDouble'},
     {'name': 'MapY', 'type': 'esriFieldTypeDouble'},
     {'name': 'TimeCreate', 'type': 'esriFieldTypeString'},
     {'name': 'TimeDispatch', 'type': 'esriFieldTypeString'},
     {'name': 'TimeArrive', 'type': 'esriFieldTypeString'},
     {'name': 'TimeClosed', 'type': 'esriFieldTypeString'},
     {'name': 'Disposition', 'type': 'esriFieldTypeString'},
     {'name': 'DispositionText', 'type': 'esriFieldTypeString'},
     {'name': 'BLOCK_ADDRESS', 'type': 'esriFieldTypeString'},
     {'name': 'Zip', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'PoliceDistrict', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'Location', 'type': 'esriFieldTypeString'}]},
   'geometry': {'geometryType': 'esriGeometryPoint',
    'spatialReference': {'wkid': 4326},
    'fields': [{'name': 'MapX', 'formats': ['x']},
     {'name': 'MapY', 'formats': ['y']}]},
   'time': {'timeType': 'instant',
    'timeReference': {'timeZone': 'UTC'},
    'fields': [{'name': 'TimeCreate',
      'formats': ['MM/dd/yyyy hh:mm:ss a']}]}}]}

manifest['datasets'][0]['geometry']['spatialReference'] = { "wkid": 102682, "latestWkid": 3452 }

file_share_folder.manifest = manifest

file_share_folder.manifest

{'datasets': [{'name': 'calls',
   'format': {'quoteChar': '"',
    'fieldDelimiter': ',',
    'hasHeaderRow': True,
    'encoding': 'UTF-8',
    'escapeChar': '"',
    'recordTerminator': '\n',
    'type': 'delimited',
    'extension': 'csv'},
   'schema': {'fields': [{'name': 'NOPD_Item', 'type': 'esriFieldTypeString'},
     {'name': 'Type_', 'type': 'esriFieldTypeString'},
     {'name': 'TypeText', 'type': 'esriFieldTypeString'},
     {'name': 'Priority', 'type': 'esriFieldTypeString'},
     {'name': 'MapX', 'type': 'esriFieldTypeDouble'},
     {'name': 'MapY', 'type': 'esriFieldTypeDouble'},
     {'name': 'TimeCreate', 'type': 'esriFieldTypeString'},
     {'name': 'TimeDispatch', 'type': 'esriFieldTypeString'},
     {'name': 'TimeArrive', 'type': 'esriFieldTypeString'},
     {'name': 'TimeClosed', 'type': 'esriFieldTypeString'},
     {'name': 'Disposition', 'type': 'esriFieldTypeString'},
     {'name': 'DispositionText', 'type': 'esriFieldTypeString'},
     {'name': 'BLOCK_ADDRESS', 'type': 'esriFieldTypeString'},
     {'name': 'Zip', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'PoliceDistrict', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'Location', 'type': 'esriFieldTypeString'}]},
   'geometry': {'geometryType': 'esriGeometryPoint',
    'spatialReference': {'wkid': 102682, 'latestWkid': 3452},
    'fields': [{'name': 'MapX', 'formats': ['x']},
     {'name': 'MapY', 'formats': ['y']}]},
   'time': {'timeType': 'instant',
    'timeReference': {'timeZone': 'UTC'},
    'fields': [{'name': 'TimeCreate',
      'formats': ['MM/dd/yyyy hh:mm:ss a']}]}}]}

# You can regenerate a manifest if you have added new data or if you have uploaded a hints file using the edit resource.
file_share_folder.regenerate()

True