Feature: Active/Standby Management

Summary

When the Feature Session Persistence is enabled, there can only be one active/providing service Drools PDP due to the behavior of Drools persistence. The Active/Standby Management Feature controls the selection of the Drools PDP that is providing service. It utilizes its own database and the State Management Feature database in the election algorithm. All Drools PDP nodes periodically run the election algorithm and, since they all use the same data, all nodes come to the same conclusion with the “elected” node assuming an active/providingservice state. Thus, the algorithm is distributed and has no single point of failure - assuming the database is configured for high availability.

When the algorithm selects a Drools PDP to be active/providing service the controllers and topic endpoints are unlocked and allowed to process transactions. When a Drools PDP transitions to a hotstandby or coldstandby state, the controllers and topic endpoints are locked, preventing the Drools PDP from handling transactions.

Usage

Enabling and Disabling Feature State Management

The Active/Standby Management Feature is enabled from the command line when logged in as policy after configuring the feature properties file (see Description Details section). From the command line:

  • > features status - Lists the status of features
  • > features enable active-standby-management - Enables the Active-Standby Management Feature
  • > features disable active-standby-management - Disables the Active-Standby Management Feature

The Drools PDP must be stopped prior to enabling/disabling features and then restarted after the features have been enabled/disabled.

Enabling Active/Standby Management Feature
 policy@hyperion-4:/opt/app/policy$ policy stop
 [drools-pdp-controllers]
  L []: Stopping Policy Management... Policy Management (pid=354) is stopping... Policy Management has stopped.
 policy@hyperion-4:/opt/app/policy$ features enable active-standby-management
 name                      version         status
 ----                      -------         ------
 controlloop-utils         1.1.0-SNAPSHOT  disabled
 healthcheck               1.1.0-SNAPSHOT  disabled
 test-transaction          1.1.0-SNAPSHOT  disabled
 eelf                      1.1.0-SNAPSHOT  disabled
 state-management          1.1.0-SNAPSHOT  disabled
 active-standby-management 1.1.0-SNAPSHOT  enabled
 session-persistence       1.1.0-SNAPSHOT  disabled

Description Details

Election Algorithm

The election algorithm selects the active/providingservice Drools PDP. The algorithm on each node reads the standbystatus from the StateManagementEntity table for all other nodes to determine if they are providingservice or in a hotstandby state and able to assume an active status. It uses the DroolsPdpEntity table to verify that other node election algorithms are currently functioning and when the other nodes were last designated as the active Drools PDP.

In general terms, the election algorithm periodically gathers the standbystatus and designation status for all the Drools PDPs. If the node which is currently designated as providingservice is “current” in updating its status, no action is required. If the designated node is either not current or has a standbystatus other than providingservice, it is time to choose another designated DroolsPDP. The algorithm will build a list of all DroolsPDPs that are current and have a standbystatus of hotstandby. It will then give preference to DroolsPDPs within the same site, choosing the DroolsPDP with the lowest lexicographic value to the droolsPdpId (resourceName). If the chosen DroolsPDP is itself, it will promote its standbystatus from hotstandby to providingservice. If the chosen DroolsPDP is other than itself, it will do nothing.

When the DroolsPDP promotes its standbystatus from hotstandby to providing service, a state change notification will occur and the Standby State Change Handler will take appropriate action.

Standby State Change Handler

The Standby State Change Handler (PMStandbyStateChangeHandler class) extends the IntegrityMonitor StateChangeNotifier class which implements the Observer class. When the DroolsPDP is constructed, an instance of the handler is constructed and registered with StateManagement. Whenever StateManagement implements a state transition, it calls the handleStateChange() method of the handler. If the StandbyStatus transitions to hot or cold standby, the handler makes a call into the lower level management layer to lock the application controllers and topic endpoints, preventing it from handling transactions. If the StandbyStatus transitions to providingservice, the handler makes a call into the lower level management layer to unlock the application controllers and topic endpoints, allowing it to handle transactions.

Database

The Active/Standby Feature creates a database named activestandbymanagement with a single table, droolspdpentity. The election handler uses that table to determine which DroolsPDP was/is designated as the active DroolsPDP and which DroolsPDP election handlers are healthy enough to periodically update their status.

The droolspdpentity table has the following columns:
  • pdpId - The unique indentifier for the DroolsPDP. It is the same as the resourceName
  • designated - Has a value of 1 if the DroolsPDP is designated as active/providingservice. It has a value of 0 otherwise
  • priority - Indicates the priority level of the DroolsPDP for the election handler. In general, this is ignore and all have the same priority.
  • updatedDate - This is the timestamp for the most recent update of the record.
  • designatedDate - This is the timestamp that indicates when the designated column was most recently set to a value of 1
  • site - This is the name of the site

Properties

The properties are found in the feature-active-standby-management.properties file. In general, the properties are adequately described in the properties file. Parameters which must be replaced prior to usage are indicated thus: ${{parameter to be replaced}}

feature-active-standby-mangement.properties
 # DB properties
 javax.persistence.jdbc.driver=org.mariadb.jdbc.Driver
 javax.persistence.jdbc.url=jdbc:mariadb://${{SQL_HOST}}:3306/activestandbymanagement
 javax.persistence.jdbc.user=${{SQL_USER}}
 javax.persistence.jdbc.password=${{SQL_PASSWORD}}

 # Must be unique across the system
 resource.name=pdp1
 # Name of the site in which this node is hosted
 site_name=site1

 # Needed by DroolsPdpsElectionHandler
 pdp.checkInterval=1500 # The interval in ms between updates of the updatedDate
 pdp.updateInterval=1000 # The interval in ms between executions of the election handler
 #pdp.timeout=3000
 # Need long timeout, because testTransaction is only run every 10 seconds.
 pdp.timeout=15000
 #how long do we wait for the pdp table to populate on initial startup
 pdp.initialWait=20000

End of Document