Updated on 7/10/2019
DR Design Guides
Failover Advanced mode Configuration - Parallel thread and failover jobs
Direct link to topic in this publication:

Failover Advanced mode Configuration - Parallel thread and failover jobs

These configurations are aimed at customers that have great than 50 policies for business reasons and require faster failover option to maintain SLA on data recovery.

Two features exist:

  1. Parallel threads - allows make writeable and resync prep to operate in parallel upto thread limit of 10.  This means 10 policies will be executed at a time for all SyncIQ steps and Eyeglass will ensure that at least 10 policies are executing at a time throughout the failover process.
  2. Parallel Access Zone or Pool Failover - Default mode of failover allowed a single access zone or ip pool to failover at one time.  If more than one failover was submitted, the 2nd, 3rd etc.. failover would wait in a queue until the 1st failover completed.   This new feature allows customers to start multiple failover jobs in parallel to accelerate the failover process when the customer has multiple assess zones configured.

Parallel threads

This mode switches to parallel policy with up to 10 threads for make writeable step and resync prep step.  This defaults to disabled and uses sequential make writeable and resync prep steps when a group of policies are involved in a failover job.  The default is sequential which is Recommended as Best practice least risk mode.     

Key differences between default sequential and parallel mode:

  1. For 8.x clusters, 50 policies can run at a time and Eyeglass will use a maximum of 10 threads allow 10 policy make writeable or resync prep commands to be sent in at once. For 7.2 clusters only 5 will execute and 5 are queued.  If one policy completes another policy is started with the goal of keeping maximum number queued at all times.  
  2. Testing has shown 3x to 4x overall time to complete make writeable improvements.  Results in production may vary.
  3. NOTE: As of release 2.0 and later any failed policy step will block any further syncIQ steps that can not be run but will continue with other policy syncIQ steps without failing the failover job.

How to enable Fast Failover parallel threads advanced mode

  1. igls adv failovermode set --parallel=true
  2. Done. The change affects all failover jobs.
  3. Disable with
  4. igls adv failovermode set --parallel=false

Parallel Failover Jobs

This feature allows multiple failover jobs of any type to be failed over in parallel to reduce failover time.  This feature still has a 10 thread limit for all failover jobs.  This can be combined with the parallel threads feature to increase each failover jobs parallelization.  Testing this in advance of a failover is mandatory step.   NOTE:  Do NOT start more than 4 concurrent failovers. This is the tested limit.  A failover job can be any type of failover with any number of policies in each failover job.  

How to enable Fast Failover parallel Access zones/IP pools advanced mode

  1. Login via ssh as admin to eyeglass
  2. sudo -s
  3. Enter admin password to become root user
  4. Type: vim /opt/superna/sca/data/system.xml
  1. Add tag to this file as per below within the <process>  section of the file
  2. Type i to enable insert mode and hit enter key to create a new blank line and past this text beow into 
  3. <run_concurrent_fofb>true</run_concurrent_fofb>
  4. :wq (colon then wq to save an quit)
  1. systemctl restart sca
  2. The feature is now enabled after the restart command above.
  3. To submit parallel access zone or IP Pool.  Use DR Assistant to start a failover job.  
  4. Close DR Assistant, re-open it and start another failover.
  5. Repeat, the above step to submit more parallel failover jobs.
  6. Monitor all failovers from DR Assistant running failover tab.
  7. NOTE:  Cluster resources may be exhausted and testing is mandatory prior to attempting a very large number of failovers.  
Copyright Superna LLC