PAM guide to High Availability (HA)

This guide is to help you understand PAM HA and how it works.

Introduction

High Availability (HA) with PAM, allows the configuration of two PAM Servers to work as a HA Pair, with one server being the active server and the other server in a standby state.

The first PAM Server is deployed as the Primary and becomes the active server. The Primary is used to configure the HA Pair and will receive all user traffic as the active server.

The second PAM Server is deployed as the Secondary and will become the standby server. The standby server will receive encrypted database replication from the active server so it is ready to take over if the active server should fail.

Failover from an active server to the standby server is a manual process and will require the ha-failover command to be run on the Secondary server. This will switch the Secondary server from standby to active so it can start receiving user traffic.

The benefits of implementing a PAM HA solution include:

Minimising the loss of data through physical streaming database replication.
Increased availability and minimal downtime as the standby can quickly be promoted and continue operations.
Easy access to the PAM User Interface using the floating IP address.

The following diagram shows the high level architecture for a PAM HA deployment.

Osirium PAM Architecture - High Availability

High Availability terms

Term	Description
HA Pair	Refers to a pair of PAM Servers that have been configured as a Primary and Secondary.
Primary	First PAM Server deployed as HA Primary.
Secondary	Second PAM Server deployed as HA Secondary.
Active	The PAM Server that is receiving all the user traffic and replicating to the standby PAM Server.
Standby	The PAM Server that is receiving replication data from the active server.
Floating IP	A floating IP is an IP address that can be used by either HA server in the same subnet, when the server is active. It is setup as an alias on the Primary PAM Server during configuration. During failover, the floating IP address is moved from the old active (Primary server) to the new active (Secondary server), enabling users to continue using a single IP address to log into the PAM User Interface.
Joining code	The joining code enables bidirectional key-based access to be established. It will be displayed on the console window of the Secondary server or can be created when replacing a Primary server.
Management Interface	The Management Interface is used to create, configure and manage a HA Pair.

HA port requirements

The following ports are required for communication between the PAM Servers in the HA Pair:

Protocol	Port	Description
TCP	22	Primary to Secondary connectivity, both directions.
TCP	5432	HA Database replication, both directions.
TCP	8443	Management Interface, both directions.

For information relating to all ports used by PAM click here.

Data replication

A HA Pair uses streaming physical replication which enables updated information on the active server to be transferred to the standby server in real-time. This allows the database on the standby server to be kept in sync with the active server.

Master encryption key

When a PAM Server is configured as the Primary server, the master encryption key of this server is very important. The master encryption key of the Primary PAM Server should be kept safe and stored in your vault as it will be required if the system needs to be recovered from a backup file.

When the Secondary server is created, there is no master encryption key. When you initialise your HA Pair, the master encryption key of the Primary PAM Server is replicated to the Secondary PAM Server.

Note

If the Secondary server is reverted back to a standalone PAM Server it will keep the MEK replicated from the Primary server.

Files that are generated when using PAM will be stored on the internal system disk (/data/osirium/filestore) of the PAM Server unless a file store (data disk or SMB share) is configured.

Using the internal system disk for generated files is NOT recommended as it will fill up the server disk space and may cause system issues.

For a PAM HA deployment we recommend an SMB share is configured to allow files created on an active server (Primary or Promoted Primary), to be accessible to either server in the HA Pair when the server is active. Files created can be managed on the Manage files page within the Admin Interface.

Warning

If an SMB share is not configured then you are at risk of losing files stored locally on the PAM Server.

An SMB file store is configured on the System settings tab within the Admin Interface. You will firstly need to select SMB share as the file store method to use and then configure the SMB share details so PAM can connect to the share and save files to that location.

Depending on your configuration, files that are generated when using PAM and which will be stored on the SMB share may include:

PAM Server backup files
PAM Server Techout files
Files created by device tasks run
Session Recordings
Session archives (if enabled)

Management Interface

The Management Interface is used to create, configure and manage a HA Pair. It is accessed via a web browser by navigating to [IP Address of the PAM Server]:8443and requires PAM Owner user access rights.

See the Management Interface Guide for more details.

System backup and recovery

To prevent data loss during system failure, it is important that regular backups are taken and kept in accordance with your corporate backup and recovery strategy.

Restoring a HA Pair will require a backup file of the Primary server and the MEK. Therefore, we recommend regular backups are taken and files saved to a file store.

The PAM Server backup task can be used to create the Osirium backup file of the active server needed to rebuild a HA Pair. The Osirium backup file can be created on an adhoc basis or it can be scheduled through a profile.

Osirium backup files will contain the following data and configuration:

Copy of the database which includes all user, device and profile configurations
Task logs
Templates
Version information
Encrypted passphrase (Only if the Backup breakglass passphrase has been configured.)
Certificates

Logs

The logging information relating to HA can be found in the following location:

HA command logs: /var/log/osirium/ha-commands.log
HA proxy logs: Available on the Logs within the Admin Interface.

If you are transitioning from a cluster to a HA Pair then these additional logs will be available:

Declustering commands: /var/log/osirium/install/decluster.log