Managing HA Server States

This section looks at server states and how they can be resolved.

• HA PAM Server States

• HA PAM Server Commands

• How To Manage Server States

HA PAM Server states

The following table describes the possible states a server in a HA Pair could be in:

State	Description
Active	Server is operational. In BAU, the Primary PAM Server is the active server. In failover, the Secondary PAM Server will become the promoted active server.
Standby	The PAM Server is receiving information from the active server. The data replicated to the standby will allow it to take over operations if the active server fails.
Demoted	Only a Primary PAM Server can be demoted. Operation has been switched to the standby server.
Failed	The PAM Server is broken and is not functional, it could be off line, have a major issue, or has been deleted.
Inactive	Server is not operational. A demoted Primary or a standby Secondary can be considered as inactive.

The HA states are monitored by PAM and can be viewed in the following:

• Admin Interface High Availability tab
• Management Interface

Admin Interface High Availability tab

Within the Admin Interface, the High Availability tab which can be found in System > System configuration, provides a high level overview of the HA Pair.

Management Interface

The Management Interface is used to manage HA configuration and execute HA commands on the PAM Servers.

It is accessed via a web browser by navigating to https://[IP Address of the PAM Server]:8443. Only a PAM user who has PAM Owner level access can log in.

Once logged in you will be presented with an SSH window.

Within the Management Interface, you can run the ha-status command to get details on your HA Pair.

HA PAM Server Commands

There are a number of commands that are used to manage your HA Pair and its states. Commands are executed through the Management Interface which is accessed via a web browser by navigating to https://[IP Address of the PAM Server]:8443.

Commands will differ depending on whether you are on the active or standby server.

The following table describes the commands available:

Command	Description
ha-initialise	Used to create a HA Pair and start replication between the PAM Servers in the HA.
ha-create-joining-code	Used to generate a joining code if one is not visible on the server console window. The joining code is required when creating a new HA Pair and to failback to a replacement Primary server.
ha-failover	Run on the standby server If the Primary fails then this command is used on the Secondary to switch it to active so it can start receiving user traffic.
ha-demote-primary	Run on the Primary server. Used to demote the Primary so the operation is switched to the standby server.
ha-failback	Run on the Secondary server If a Primary failed and has been recovered, the Secondary server can hand back operations to the Primary server. The Primary will again become the active server and the Secondary will go back to being a standby server.
ha-status	Provides server and HA status information.
ha-make-standalone	Can be run on the Primary or Secondary to revert a HA Server back to a standalone PAM Server and stop replication.
ha-initialise --ha-ip=[floating ip]	When a HA Pair is created using the ha-initialise command, you are given the option to add a floating ip address. If this is left blank during the initial setup, then it can be configured later using this command.

How to manage server states

• Scenario 1: Failed Active Primary
• Scenario 2: HA Split Brain
• Scenario 3: Failed Standby Secondary
• Scenario 4: Both HA Servers Fail

Scenario 1: Failed active Primary

This scenario illustrates a BAU HA Pair against a failed active Primary. When an active Primary fails the server maybe offline or still online but not working correctly. When a active Primary goes offline, it can't be demoted. An active Primary that is still online but is having issues can still be demoted when you failover to the standby server. There is no database replication during this period.

To recover a HA Pair from a failed active Primary will require the following steps:

1. On the standby Secondary server run ha-failover command. The standby server will be promoted to active and the Primary demoted if it is available.

2. Deploy and configure a new PAM Server as a replacement Primary server.

3. On the Secondary server run ha-failback command. The Primary will again become the active Primary server and the Secondary will be demoted back to standby. Replication will restart between the Primary and Secondary.

For detailed instructions on how to recover from a failed Primary, click here.

Scenario 2: HA split brain

This scenario illustrates a BAU HA Pair against a split brain. A HA Pair can end up in a split brain scenario if both servers in the HA Pair have a status of active. There is no database replication during this period.

To recover a HA Pair from a split-brain scenario will require the following steps:

1. On the Primary server, run the ha-demote-primary command. The Primary server will be demoted and will be on standby.

2. On the Secondary server, run the ha-failback command. The Primary will become the active server again and replicate to the Secondary. The Secondary server will be demoted back to standby.

Note

If any configuration changes were made on the Primary they will be lost when the failback command is run on the Secondary.

Scenario 3: Failed standby Secondary

This scenario illustrates a BAU HA Pair against a failed standby Secondary. When a standby secondary fails, there is no database replication but users can continue as normal.

To recover a HA Pair from a failed standby Secondary will require the following steps:

1. Deploy a new PAM Server as HA Secondary.

2. On the Primary server, run ha-initialise and configure the new Secondary server details to create the HA Pair. The joining code of the Secondary server will be required which can be found on the console window.

3. Secondary set to standby.

Scenario 4: Both HA servers fail

This scenario illustrates a BAU HA Pair against both HA servers failing at the same time. In this case the Primary server will have to be restored from a backup file.

To recover a HA Pair from a failed Primary and Secondary will require the following steps:

1. Deploy two PAM Servers.

2. During the configuration of the first server, select the Restore a backup installation type. The HA MEK will be required. This will become the Primary server.

3. Configure the second server as a HA Secondary.

4. On the Primary server ha-initialise to recreate the HA-Pair. The Primary server will be set to active. The Secondary server will be set to standby.