Managing your GIS servers
At version 1.2 the ArcGIS API for Python introduces a new server submodule under the gis.admin module that allows administrators to manage the ArcGIS servers powering their ArcGIS Enterprises. ArcGIS Enterprise includes several software components that are designed to work together. A basic ArcGIS Enterprise deployment can consist of the following:
Refer to ArcGIS Enterprise deployment patterns to learn more about different deployment patterns for ArcGIS Enterprise. No matter which pattern your deployment is, the server module can be used to access and manage it. The diagram below represents the main classes and methods available in the server module
To start with, import the Python API and connect to your GIS as an administrator.
Note: To make use of the server module, you need to connect it using an account with administrative privileges. The server module is only applicable if your GIS is an instance of ArcGIS Enterprise and would not work for ArcGIS Online.
from arcgis.gis import GIS
gis = GIS("http://siteurl.mysite.com/portal", "username")
Accessing the servers behind your GIS¶
You can get the list of all servers hosted or federated to your Enterprise by calling the list() method from admin.servers object.
gis_servers = gis.admin.servers.list()
gis_servers
server1 = gis_servers[0]
type(server1)
Once you access your Server object, you can perform operations such as managing data stores, querying logs, accessing services, publishing new services etc. as explained in the rest of this guide
Validate your servers¶
You can quickly check if your servers are responding to your Enterprise by calling the validate() method. If you get back False, you may need to check one or more of your servers.
gis.admin.servers.validate()
Managing the services on your server¶
Getting the list of services¶
One of the important tasks you can perform on a server is monitoring the services running on it and publishing new ones. You can list the services by calling list() method from the Server.services object. By default, you get the services running in root folder.
server1.services.list()
To get the services running in a different folder, specify its name as an argument. To get the list of folders on your server, call the folders property
server1.services.folders
List the services running in System folder
hosted_services = server1.services.list(folder='System')
#print the top 5 as a sample
hosted_services[0:4]
Checking if a service exists¶
To check if a service exists on your server, call the exists() method and specify the folder name, service name and type. You can also use this method to verify if a folder exists on the server.
# check if a FeatureService called 'Ports' exists in Hosted folder
server1.services.exists(folder_name='Hosted', name='Ports', service_type='FeatureServer')
Managing service folders¶
You can add or remove folders on your server by calling create_folder(), delete_folder().
Note: These folders are different from folders found on user's 'my contents'. To access those folders see ConentManager.create_folder and User.folders
server1.services.create_folder('crime_analysis')
You can delete this folder by calling delete_folder()
server1.services.delete_folder('crime_analysis')
Administering services¶
If you noticed earlier, calling services.list() returns you a list of Service objects. Using a Service object, you can start, stop, delete, edit, rename and query the properties of a service.
# access the 'SampleWorldCities' service running on root folder
service1 = server1.services.list()[-1]
type(service1)
Query the properties property to get details on the service
service1.properties
You can also query fine-grained information such as the capabilities enabled on the service
service1.properties.capabilities
You can get the id of one or more Items corresponding to this service on the Enterprise as shown below:
service1.properties.portalProperties.portalItems
Getting status on a service¶
Calling the status property returns a dictionary with the real time state of the service.
service1.status
Starting and stopping a service¶
Call the start() and stop() methods to start and stop a service.
service1.stop()
#call the status to confirm
service1.status
#start the service and confirm
service1.start()
service1.status
Publishing new services¶
You can publish new services to your server by calling publish_sd() method from Server.services object.
Note: You need a service definition file to publish a service. You can create SD files using ArcGIS Desktop and Pro. To automate the creation of SD files from your map or project files, refer to ArcPy documentation
server1.services.publish_sd(sd_file_path='/Users/atma6951/Documents/GIS_data/SanDiego_maps.sd',
folder=None)
Publishing a service to the server will create an item on the Enterprise. In this case, the service was called 'AGSMS_SD', hence the name of the item.
new_item = gis2.content.search('owner:atma.mani', item_type = 'Map Service')[0]
new_item
Map Image Layer by atma.mani
new_item.url
Renaming a service¶
As ArcGIS administrators, you must be familiar that the name of the service forms its URL. You can rename a service, however, this operation should be used carefully since it would break all the layers and maps that refer to this service by its URL
# search for the service
new_service = server1.services.list()[0]
type(new_service)
Give this service a descriptive name. Service names cannot contain spaces or special characters.
new_service.rename('SanDiego_boundary')
Renaming will create a new item on the Enterprise corresponding to the new URL
gis2.content.search('owner:atma.mani', 'Map Service')
The first Item is now stale and should be deleted by you. The second Item corresponds to the renamed service. You can access and print its url property to verify:
renamed_item = gis2.content.search('owner:atma.mani', 'Map Service')[1]
renamed_item.url
Thus, you can rename services, or change their URLs using the Python API. You cannot however, use rename() to move a service from one folder into another folder on the server after it is published.
Deleting a service¶
To delete a service, simply call the delete() method on the Service object.
#Delete the San Diego service. Since it was renamed, you need to search for it
service_list = server1.services.list()
service_list
service_list[-1].delete()
Managing server data stores¶
ArcGIS Data Store is an application that lets you easily configure data storage for hosting and federated servers used with your Enterprise. ArcGIS Data Store provides you with a convenient setup and configuration experience that creates the following different types of data stores.
- Relational data store -- Stores your portal's hosted feature layer data, including hosted feature layers created as output from spatial analysis tools run in the portal
- Tile cache data store -- Stores caches for your portal's hosted scene layers
- Spatiotemporal big data store -— Archives real-time observational data that you can use with an ArcGIS Server running ArcGIS GeoEvent Server that is federated with your portal; also stores the results generated using ArcGIS GeoAnalytics Server tools.
Using the Server.datastores object, you can list, add, remove datastore items as shown below:
Listing the data stores registered with the server¶
Calling the datastores.list() method returns you a list of Datastore objects.
dstores = server1.datastores.list()
len(dstores)
dstore1 = dstores[0]
type(dstore1)
Print the path for each of these Datastore objects.
for dstore in dstores:
print(dstore.properties.path)
Registering new data stores with the server¶
You can add or register new data stores with your server by calling the corresponding add method such as add_folder(), add_database(), add_big_data(). To know what types of data stores are supported by your server, query the data_items property:
server1.datastores.data_items
Add a folder data store
ca_datastore = server1.datastores.add_folder(name='CA_earthquakes',
server_path='\\teton\atma_shared\datasets\californiapoints')
ca_datastore
type(ca_datastore)
The add_folder() method returns you a Datastore object. This object can be used to inspect, modify or delete / unregister that data store entry. You can inspect the properties of the data store by querying the properties property.
ca_datastore.properties
Validating data stores¶
You can validate the data store registered with your server to ensure your server can continue to access them.
ca_datastore.validate()
You can validate all the data stores by calling the validate() method from the Server.datastores object
server1.datastores.validate()
Removing data store entries¶
To remove a data store, simply call the delete() method from the corresponding Datastore object:
ca_datastore.delete()
Querying server logs¶
ArcGIS Server records events that occur, and any errors associated with those events, to logs. Logs are an important tool for monitoring and troubleshooting problems with your site. Information in the logs will help you identify errors and provide context on how to address problems. Refer about server logs to learn more about your ArcGIS Server logs, setting different logging levels etc.
You can access a server's logs using the logs property of the Server object.
server1.logs.settings
Filtering and querying server logs¶
Using the query() method, you can filter and search for server logs. Refer to the query API ref doc for all the arguments supported. In the example below, logs for the previous 10 days is searched.
import datetime
import pandas as pd
now = datetime.datetime.now()
start_time = now - datetime.timedelta(days=10)
start_time
You can pass a Python Datetime object to the time arguments.
recent_logs = server1.logs.query(start_time = start_time)
#print a message as a sample
recent_logs['logMessages'][0]
You can construct a Pandas DataFrame from the query result and visualize the logs as a table:
log_df = pd.DataFrame.from_records(recent_logs)
log_df.head(5) #display the first 5 records
You can quickly check for errors reports as shown below:
server1.logs.count_error_reports()
Monitoring server usage¶
ArcGIS Server records various service statistics, such as total requests, average response time and timeouts. Administrators and publishers can use this information to monitor service activity to better understand how clients are using services. For example, monitoring server statistics help you answer questions such as:
- What is the total number of requests that my ArcGIS Server site handled during the past week?
- How was the service request load distributed during the past month?
- How are my services performing on an hourly basis?
- What was the maximum number of service instances used at any given time for a particular service?
To learn more about this, refer to about server statistics help page. To access this information, use the Server.usage property:
server1.usage.settings
Using built-in reports¶
By default, when you install ArcGIS Server, you get three pre-configured reports. These reports are based on services that are already running on your server. You can access these and any new reports you create by querying the list() method.
usage_reports = server1.usage.list()
usage_reports
You get back a list of Report objects. Print the names of the reports:
for r in usage_reports:
print(r.properties['reportname'])
Querying maximum reponse times for the last 7 days¶
max_resp_report = usage_reports[0]
#Run the query
data = max_resp_report.query()
#display the keys in the report query response
data['report'].keys()
Plot the response time in a bar chart
from datetime import datetime
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
%matplotlib inline
#store reponse times in Y axis
data_y = data['report']['report-data'][0][0]['data']
#convert dates to readable dates and store in X axis
data_x = [pd.to_datetime(datetime.fromtimestamp(d//1000)) \
for d in data['report']['time-slices']]
df = pd.DataFrame(list(zip(data_x, data_y)), columns=["date", "count"])
q = df['count'].isnull() # change NaN values to 0
df.loc[q, 'count'] = 0
df.index = df['date']
df['count'] = df['count']
ax = df['count'].plot(kind='bar', x=df['date'])
ticklabels = ['']*len(df.index)
ticklabels[::4] = [item.strftime('%b %d') for item in df.index[::4]]
ax.xaxis.set_major_formatter(ticker.FixedFormatter(ticklabels))
ax.set_title('Maximum reponse time in the last 7 days')
ax.set_ylabel('Time in seconds')
plt.gcf().autofmt_xdate()
plt.show()
Creating Quick Reports¶
Quick reports are useful for administrators who want to get a one-shot set of data. This information is generated on the fly on ArcGIS Server, and allows for report generation for services or server information that may not be monitored all the time.
Note: On the fly reports that are deleted right after data is returned
The quick report supports various time spans:
- LAST_DAY - 24 hours from your current time
- LAST_WEEK - 7 days from your current date
- LAST_MONTH - 31 days from your current date
- LAST_YEAR - 365 days from your current date
Reporting Queries are made by specifying the resource you want to monitor. For example, administrators who want to monitor a specific folder, HOSTED, would set the query to: services/Hosted. The default is to monitor all services.
Quick report metrics are:
- RequestCount - the number of requests received
- RequestsFailed - the number of requests that failed
- RequestsTimedOut - the number of requests that timed out
- RequestMaxResponseTime - the maximum response time
- RequestAvgResponseTime - the average response time
- ServiceActiveInstances - the maximum number of active (running) service instances sampled at 1 minute intervals, for a specified service
Get the total number of requests in the last 1 year
data = server1.usage.quick_report(since="LAST_MONTH", metrics="RequestCount")
data['report'].keys()
Plot the results in a bar chart
#store reponse times in Y axis
data_y = data['report']['report-data'][0][0]['data']
#convert dates to readable dates and store in X axis
data_x = [pd.to_datetime(datetime.fromtimestamp(d//1000)) \
for d in data['report']['time-slices']]
#create a Pandas DataFrame from the report
df = pd.DataFrame(list(zip(data_x, data_y)), columns=["date", "count"])
q = df['count'].isnull() # change NaN values to 0
df.loc[q, 'count'] = 0
df.index = df['date']
df['count'] = df['count']
#plot as a bar chart
ax = df.plot(kind='bar', x=df['date'])
ticklabels = ['']*len(df.index)
ticklabels[::4] = [item.strftime('%b %d') for item in df.index[::4]]
ax.xaxis.set_major_formatter(ticker.FixedFormatter(ticklabels))
plt.gcf().autofmt_xdate()
plt.show()
Accessing the machines powering your servers¶
You can access the machines powering your GIS servers using the Server.machines property. By calling the list() method, you get back a list of Machine objects:
machine_list = server1.machines.list()
machine_list
Access the properties of a machine using properties.
m1 = machine_list[0]
m1.properties
Use the status property to verify the state of your machine.
m1.status
Get the list of SSL certificates:
m1.ssl_certificates
Download existing certificates or import new ones to your server using the appropriate methods:
m1.export_certificate(m1.ssl_certificates['certificates'][0])
all_dir = server1.system.directories().all()
all_dir[0]
Print the name and path on disk for each of these server directories:
for fld in all_dir:
print(fld.properties.name + " : " + fld.properties.physicalPath)
By calling the clean() and edit() methods on a Directory object, you can clean the contents or designate a different directory path for the server.
Monitoring server licenses and extensions¶
Use the licenses property of Server.system object to get a dictionary of various licenses and extensions available on the server.
server1.system.licenses.keys()
Find the license level of the current server:
server1.system.licenses['edition']
Print all the extensions and their expiration:
from datetime import datetime
for extn in server1.system.licenses['features']:
#convert expiration to a readable format
d = datetime.fromtimestamp(int(extn['expiration']/1000))
print("{:50s} {}".format(extn['displayName'], d.isoformat()))