PAM Server inline full cluster upgrade procedure

This section looks at how to inline upgrade a cluster deployment.

Prerequisites

Before starting your inline full cluster upgrade make sure the following prerequisites are met:

Prerequisite	Description
Upgrade path	Review the Cluster upgrade path to ensure you are applying the correct upgrade method based on the version you are upgrading from and to.
Hardware & Software	Ensure the correct resources are available before deploying. The following outlines the hardware and software requirements.
Templates	When upgrading to v7.5.1 and above, an upgrade will be required to the latest template bundle to ensure templates continue to work. This may need to be done before the upgrade. The steps required to perform the upgrade will depend upon the template. For details on the templates affected, and how to upgrade them click here.
Ports	Ensure TCP ports 443 and 9002 are open as they are required for the client-side and PAM UI. TCP 2379 (etcd, i.e. key-value store), 2380 (etcd, i.e. key-value store), 2390 (cluster setup API), 2391 (cluster delegation API), 5432 (postgres, i.e. database) For a full list of ports used by the PAM Server click here.
Software downloads	Download the latest upgrade kit click here.
Recent backup	We recommend that you have a recent Osirium backup of the leader as well as VM Level backup or Snapshot of all nodes in the cluster.
No active user connections	Ensure there are no active user connections on any nodes.
Regenerate Account Credentials for all devices tasks	Within the Admin Interface, disable all scheduled Regenerate Account Credentials for all devices tasks within profiles.
Osirium Support account	Within the Admin Interface ensure the Osirium Support account has been enabled and a password set.
Network time protocol (NTP)	The clocks of all nodes must be within two seconds drift of each other. The PAM Server OVA is preconfigured with public ntp.org NTP servers but these can be changed to your internal corporate servers (if required) by clicking here.
Node identifiers	Nodes are identified by their address which can either be a fully qualified domain name (FQDN) (i.e. clusterleader.companyABC.net) or an IP Address. Cluster nodes communicate with each other using their assigned address, therefore the address must be unique to allow a node to resolve the address of other nodes. If you wish to use FQDNs then the names must resolve to a local address on the node before the installation can continue. All nodes must be able to resolve all FQDNs of all other nodes.
Server not NATted	If you wish to run nodes on premise and in the cloud, they must be able to communicate with each other bidirectionally using their given IP address (for example through a VPN) and not be NATted.
.local DNS domains	If you are using .local DNS domains, ensure matching records have been entered in the DNS Search Suffixes.

Upgrade procedure

The diagram provides a high-level overview of the process for upgrading all nodes in the cluster.

Full cluster upgrade procedure flowchart

Upgrade Full Cluster

Upgrade steps

Open a file transfer tool of your preference and copy the upgrade kit onto a node in your cluster using the osirium_support account.
Open the PAM Server Console window, then press ALT + F2. The server login prompt appears.

Note
Alternatively, you could use an SSH connection to the PAM Server.
Enter osirium_support at the login prompt and press ENTER.
Enter the password of the osirium_support account and press ENTER.
Extract the upgrade kit copied to the server using the following command:

sudo bash Osirium_PAM_Server_vA.B.C_upgrade.bin

Where A.B.C is the version you are upgrading to.
When the kit has been extracted, type the command specified on the screen and press ENTER.
Press ENTER when prompted to start the setup and configuration.
The EULA screen will be displayed. Press ENTER once you have read and accepted.
As you have a clustered environment, this node will now wait for all remaining nodes in the cluster to reach this stage before continuing the upgrade.
Repeat the above steps for each node in your cluster.
The cluster co-ordinator will manage the process of performing the upgrade one node at a time and it will keep you updated with its progress as to the number of nodes remaining to be upgraded.
Once all nodes have been upgraded the cluster co-ordinator will automatically reboot each of the nodes one at a time.

Post upgrade tasks

Once the full cluster upgrade has successfully completed, logon to Osirium PAM and check the following before allowing users to reconnect:

Post upgrade task	Description
Trigger AD audit	Before opening any device connections that use an Active Directory account, an audit needs to be manually triggered on all provisioned Active Directories. You can do this by right clicking the Active Directory on the Manage Active Directory page, and select `Trigger audit` from the menu. This will allow additional fields on the Active Directory account page to be populated.
Check device states	Check device status to ensure they are running successfully.
Check user connections	Check users can connect to devices.
Re-enable scheduled tasks	Re-enable scheduled Regenerate Account Credentials for all devices tasks.
Backup of all nodes	Take an Osirium backup of the leader and a VM level backup of all nodes. See Backup & Recovery for further details.
Upgrade other components	Use the PAM Component Compatibility Matrix to check if other PAM components need updating inline with the version you are upgrading to and upgrade as appropriate.
Check cluster health	Check the System Configuration -> Clustering tab of the Leader node to ensure all nodes are displayed and the Cluster status is shown as Healthy. See Clustering tab for further details. NOTE: If you are using .local search domains, because all followers will have lost their DNS suffix settings, there will be a temporary period during which the cluster will appear as being in a Degraded state, until the DNS suffix setting has been restored on all followers.