Restoring a leader using an Osirium backup file

This section looks at how to restore a failed leader node. When a leader node fails, it can be restored using an Osirium backup file but requires all follower nodes to be recreated from new and joined to the cluster. The following is covered in this section:

Prerequisites

Before starting your restore of the leader node make sure the following prerequisites are met:

Prerequisite	Description
Hardware & Software	Ensure the correct resources are available before deploying. The following outlines the hardware and software requirements.
Ports	Ensure TCP ports 443 and 9002 are open as they are required for the PAM UI and PAM Client. TCP 2380 (etcd, i.e. key-value store), 2390 (cluster setup API), 2391 (cluster delegation API), 5432 (postgres, i.e. database) For a full list of ports used by the server click here .
Software downloads	Download the same Osirium PAM version the cluster was deployed onto. To download the latest PAM Server software release package for deployment into your infrastructure, click here To download earlier versions of the PAM Server software packages, please contact Osirium support by clicking here.
*Recent Leader Osirium* backup file**	Ensure the leader Osirium backup file is available and accessible. See Data and Configuration backup for more information on backup file requirements.
Master Encryption Key	Ensure you have the Master Encryption Key (MEK) of the leader node.
Unmount external drives	If you have any external drives configured then unmount from the existing leader node.
Power off all follower nodes	Power down all the follower nodes in the cluster.

Restore from an Osirium backup file procedure

The diagram provides a high-level overview of the process for restoring your PAM Server leader node using the stored Osirium backup file and then rebuilding the followers and joining them to the restored cluster leader.

Restore a leader node using a backup procedure

Warning

Before starting your restore make sure that all Prerequisites have been met.

Steps to restore a leader node

The following steps are required to restore a cluster using the Osirium backup file of a leader node:

Deploying the PAM Server. You will firstly need to deploy a new PAM Server to install your Osirium backup file onto. Deployment into the different supported infrastructures may vary, therefore click on a link below to be navigated to the correct deployment steps.
Open the console window of the newly deployed PAM Server.
Within the Console window, press ENTER when prompted to start the setup and configuration.
Read and accept the EULA to continue.
Within the Configure Networking screen, configure the following server settings. Press TAB to navigate between the fields.
- IP Address: Enter the IP Address which will be used to connect to the server.
- Netmask: Enter the network mask.
- Gateway: Enter the network default gateway IP address.
- Primary DNS: Enter the network primary DNS IP address.
- (Secondary DNS): Enter the secondary DNS IP address if relevant, else leave blank.
- (DNS Suffixes): Enter the DNS Suffixes. Multiple entries can be separated with a comma, else leave blank.
Once completed TAB down to the OK button and press ENTER.
When you get to the PAM Server Restore screen, SFTP onto the virtual appliance using the details shown on the screen.
Copy the leader node Osirium backup file you want to restore. Once successfully copied, the screen will update and the copied backup file will appear in the list.
Select the Osirium backup file and press ENTER. When prompted TAB to the OK button and press ENTER to confirm the restore.
Enter the leaders Master Encryption Key (including dashes). Select OK and press ENTER.
Within the Enter a hostname window, enter a name to identify the server.
TAB down to the OK button and press ENTER.
Enter the FQDN (all in lowercase) or IP Address which will be assigned.

Warning

If your Osirium PAM deployment will be using the Mesh functionality which allows the PAM Server to push a copy of its Osirium backup files to a secondary PAM Server, then you must enter an FQDN and NOT an IP Address.

If the following error occurs then make sure that the hostname can be resolved and check if it has been included in the DNS A records - see Prerequisites.

If the following errors occur then make sure that the entered hostname / address are correct and press ENTER to re-enter.
TAB down to the OK button and press ENTER.
Set a password for the Primary SuperAdmin account. The username (SuperAdmin) and the password will be used later to log into PAM.
TAB down to the OK button and press TAB.
Confirm the primary SuperAdmin account password.
TAB down to the OK button and press TAB. Wait while the system is configured and restored with the backup file.
Logon to PAM using the new FQDN or IP Address set.

Post leader node restore tasks

Once the Osirium backup has successfully installed, you will need to run through the following tasks before continuing to add any follower nodes:

Post upgrade task	Description
Remount the external drives	If you had any external drives configured then remount to the new leader node.
Trigger AD audit	Before opening any device connections that use an Active Directory account, an audit needs to be manually triggered on all provisioned Active Directories. You can do this by right clicking the Active Directory on the Manage Active Directory page, and select `Trigger audit` from the menu. This will allow additional fields on the Active Directory account page to be populated.
Check device states	Check device status to ensure they are running successfully.
Check user connections	Check users can connect to devices.
Backup node	Take an Osirium backup and VM level backup of the leader node. See Backup & Recovery for further details.
Add followers	Add your follower nodes using the below instructions.

Steps to restore follower nodes

The following steps are required to join new followers to the cluster:

Deploying the PAM Server: You will firstly need to deploy a new PAM Server for each follower. Deployment into the different supported infrastructures may vary, therefore click on a link below to be navigated to the correct deployment steps.
- Deploy using VMWare vSphere
- Deploy using Microsoft Azure
- Deploy using Microsoft Hyper-V
- Deploy using Amazon Web Services
Downloading the cluster joining bundle: To join a follower node to a cluster will require the cluster joining bundle which can only be downloaded from the leader node.
Log onto the leader node and open the Admin Interface.
Within the Admin Interface navigate to System configuration > Clustering tab.

Note

If this is first time you are adding a follower node to create a cluster then the leader node will be shown as a standalone.
Click the DOWNLOAD CLUSTER JOINING BUNDLE button.
Within the Download cluster joining bundle? window take note of the message and then click the YES, DOWNLOAD button.
Configuring the follower node: Navigate to the newly deployed PAM Server and open the Console window.
Within the Console window, press ENTER when prompted to start the setup and configuration.
Read and accept the EULA to continue.
Within the Configure Networking screen, configure the following server settings. Press TAB to navigate between the fields.
- IP Address: Enter the IP Address which will be used to connect to the server.
- Netmask: Enter the network mask.
- Gateway: Enter the network default gateway IP address.
- Primary DNS: Enter the network primary DNS IP address.
- (Secondary DNS): Enter the secondary DNS IP address if relevant, else leave blank.
Once completed TAB down to the OK button and press ENTER.
You are creating a new server so press ENTER to Continue without restoring a backup.
Within the Enter a hostname window, enter a name to identify the new server within your cluster.
TAB down to the OK button and press ENTER.
Enter the FQDN (all in lowercase) or IP Address which will be assigned to the node. The configuration entered here is used to communicate between nodes.

If the following error occurs then make sure that the hostname can be resolved and check if it has been included in the DNS A records - see Prerequisites.

If the following errors occur then make sure that the entered hostname / address are correct and press ENTER to re-enter.
TAB down to the OK button and press ENTER.
When you get to the PAM Cluster Setup screen you need to upload the downloaded Cluster joining bundle. Use the SFTP credentials provided on the Console screen to log onto the server and copy the cluster joining bundle file. Once it is successfully copied it will appear in the list.
Select the cluster joining bundle file and press ENTER.
Within the Are you sure? window press ENTER to continue.
You will be asked to enter the Master Encryption Key of the leader node.
TAB down to the OK button and press ENTER.
If the Master Encryption Key has been successful you will be notified that the process may take 5 minutes, press ENTER to continue.
Once the setup is complete the following will be displayed on the Console window. Your follower node has been successfully joined to the cluster. You will also see that the Clustering tab on the Admin Interface has been updated.
Repeat for additional follower nodes to be added.

Post follower node restore tasks

Once all the followers have been successfully joined to the cluster, you will need to run through the following tasks before allowing users to reconnect:

Post upgrade task	Description
Trigger AD audit	Before opening any device connections that use an Active Directory account, an audit needs to be manually triggered on all provisioned Active Directories. You can do this by right clicking the Active Directory on the Manage Active Directory page, and select `Trigger audit` from the menu. This will allow additional fields on the Active Directory account page to be populated.
Check device states	Check device status to ensure they are running successfully.
Check user connections	Check users can connect to devices.
Backup of all nodes	Take an Osirium backup of the leader and a VM level backup of all nodes. See Backup & Recovery for further details.
Delete old follower nodes	Delete the old follower node but keep the VM level backup files for auditing requirements.
Provide new connection details	Provide users with new hostname connection details as they will have changed.