Update (Upgrades Configuration)

Description

The update operation allows administrators to define or modify the configuration used for pods during upgrades or updates. The upgrade job configuration includes a pod placement policy, with support for container resource allocations being added at ArcGIS Enterprise on Kubernetes 12.0. Once the configuration has been updated, it will be applied whenever an upgrade job is created.

Pod placement policy

The pod placement policy may consist of a node affinity and tolerations. Node affinity allows for pods to be scheduled to nodes that have matching label key-value pairs, whereas tolerations allow for pods to be scheduled on tainted nodes that would otherwise repel them if not for matching toleration and taint key-value pairs. For more information on configuring the nodeAffinity and tolerations properties, see the Node affinity and Tolerations sections below.

Node affinity

Before establishing node affinity, a label must be placed on the node. A label is comprised of key-value pairs (key=value) that are used to attract pods to be scheduled on that node. This is done by configuring the nodeAffinity for the upgrade job's pods to use the same label key and values information as the node. Below is the general JSON syntax for nodeAffinity:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
"nodeAffinity": {
  "<requiredDuringSchedulingIgnoredDuringExecution | preferredDuringSchedulingIgnoredDuringExecution>": {
    "nodeSelectorTerms": [
      {
        "matchExpressions": [
          {
            "key": "<label key>",
            "operator": "<In | NotIn | Exists | DoesNotExist>",
            "values": [
              "<label value>"
            ]
          }
        ]
      }
    ]
  }
}

The operator property shown above specifies the relationship between nodeAffinity properties and the node’s labels. The scheduler uses the key, values, and operator information, along with whether matching values are required or preferred, to determine where the upgrade job’s pods are scheduled.

For example, a node receives the following label:

1
kubectl label nodes exampleNode sampleLabel=sampleValue

and an administrator wants to require that the key (sampleLabel) and value (sampleValue) information is applied to the the pods so they can be scheduled to the specific node. To achieve that outcome, the nodeAffinity property would need to be configured with the exact key and value information applied to the node, as well as setting the requiredDuringSchedulingIgnoredDuringExecution property:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
"nodeAffinity": {
  "requiredDuringSchedulingIgnoredDuringExecution": {
    "nodeSelectorTerms": [
      {
        "matchExpressions": [
          {
            "key": "sampleLabel",
            "operator": "In",
            "values": [
              "sampleValue"
            ]
          }
        ]
      }
    ]
  }
}

Node affinity ensures that the scheduler will attempt to schedule pods with their matching nodes. However, it may not prevent other pods that do not have matching key-value pairs from being scheduled to label nodes. Node affinity will also not remove preexisting pods that have been scheduled to the node before the label was added. Administrators wanting to repel all pods that are not designated for the upgrade job may also want to taint the node.

Tolerations

Similarly to node affinity, tainting a node requires a key-value pair, as well as an effect, to be applied to a node. Applying a taint to a node ensures that all pods that do not tolerate the node are repelled from it and are scheduled to other available nodes instead. For a pod to tolerate a tainted node, the key, values, and effect properties under tolerations must either match the tainted values exactly or have only the key value be present in the tolerations property. Below is the general JSON syntax for tolerations:

1
2
3
4
5
6
7
8
"tolerations": [
  {
    "effect": "<NoExecute | NoSchedule | PreferNoSchedule>",
    "key": "<taint key>",
    "operator": "<Exists | Equal>",
    "value": "<taint value>"
  }
]

The table below defines the different values that can be assigned to the effect and operator properties:

To demonstrate how taints and tolerations interact, consider an administrator that wants to taint a node so that only tolerant pods will be scheduled to the node, and that all preexisting pods on the node are expelled if they are not tolerant. To achieve this, a taint must first be applied to the node:

1
kubectl label nodes exampleNode sampleKey=sampleValue:NoSchedule

The exact values of the key (sampleKey), value (sampleValue), and effect (NoSchedule) properties used to taint the node would need to used to configure the toleration property:

1
2
3
4
5
6
7
8
"tolerations": [
  {
    "effect": "NoSchedule",
    "key": "sampleKey",
    "operator": "Equal",
    "value": "sampleValue"
  }
]

Tainting a node and assigning tolerations to the upgrade job's pods ensures that the node is reserved for the pods that need the node’s specialized resources, and that other pods that do not need certain requirements can run on standard nodes.

Container resource allocations

Starting at 12.0, you can define the minimum and maximum memory and CPU resource allocations for each container by configuring the containers array. Below is the general JSON syntax for containers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
"containers": [
  {
    "name": "main-container",
    "resources": {
      "memoryMin": "128Mi",
      "memoryMax": "8Gi",
      "cpuMin": "0.25",
      "cpuMax": "4",
      "customResources": {}
    }
  },
  {
    "name": "fluent-bit"
  }
]

Request parameters

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
upgradeConfig={
  "upgraderJobSpec": {
    "podPlacementPolicy":{
      "nodeAffinity": {},
        "tolerations": [
          {
            "effect": "NoSchedule",
            "key": "upgrader",
            "operator": "Equal",
            "value": "true"
          }
        ]
      },
      "containers": [
        {
          "name": "main-container",
          "resources": {
          "memoryMin": "512Mi",
          "memoryMax": "16Gi",
          "cpuMin": "1",
          "cpuMax": "8",
          "customResources": {}
        }
      },
      {
        "name": "fluent-bit"
      }
    ]
  }
}

Example usage

The following are sample POST requests for the update operation, showing the changes to requests before and after support for container resource allocations was added at 12.0:

1
2
3
4
5
6
POST /<context>/admin/system/upgrades/config/update HTTP/1.1
Host: organization.example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: []

upgradeConfig={"upgraderJobSpec":{"podPlacementPolicy":{"nodeAffinity": {},"tolerations": [{"effect": "PreferNoSchedule","key": "upgrader","operator": "Equal","value": "true"}]},"containers": [{"name":"main-container"},{"name": "fluent-bit"}]}}&f=pjson&token=aOjrLVEnRpuwfgK-G-PGdyWUQhpJDlGVAt70Z3vrHd1mihME8yx8E1EgRCoIpl3mLm2DSDVjFVbyFeQVD_Sp6DrXAIrwED7NYmMTF_GkLurVWAe-MBROTxqu5781_8qj5S_3tqoINj6OmjG8zfeztCKTUCbV-xrsNp5-50g2-UCX62H_qT0r1fIgqh_KFsFF9f3cC9kzWum47eRH9PmmZHdKGN3XjY3GghswTGCELNJbEQ_q84sz8zYE5q6jx8Cl

JSON Response example

1
{"status": "success"}