Troubleshooting Enhanced Failover

Issue

Resolution

SCADA Failover feature is not enabled in the license for this node.

This error found in ScadaRoleMgr.log indicates that your key does not support the Enhanced Failover feature. Check your key in the License Viewer, and contact GE to purchase an upgrade or replace a defective key.

You will also get an error message at iFIX startup indicating the SCADA failover feature is not enabled in the license for your node.

You must purchase the additional Enhanced Failover option (SCADA Failover) for all SCADA nodes if you plan to use SCADA Failover in iFIX.

The following errors appear:

PDB Sync Loadable Block <blockname> Not defined locally.

PDB Sync Loadable Block <blockname> Not defined on remote node.

PDB Sync Loadable Block Mismatch locally <blockname> version <ver> remote <blockname> version <ver>.

These errors indicate that your loadable block configurations on one or both nodes are not configured properly. Loadable block configurations must be the same on both primary and secondary nodes. Use the BTKCFG utility on both SCADA nodes to exactly match your loadable block configurations. Loadable blocks are also referred to as Database Dynamos.

Both SCADAs are active (as displayed in ScadaSyncMonitor, or as indicated by messages in alarm services destinations).

Verify your Ethernet connection being utilized for SCADA Synchronization (preferably this is a dedicated LAN connection).

On both SCADAs, in the Task Manager, verify that ScadaSync.exe and ScadaRoleMgr.exe are listed.

On both SCADAs, in the SCU, check your Failover configuration, start the ScadaSyncMonitor (ScadaSyncMonitor.exe is found in the iFIX install folder) and check that the transport is configured for the appropriate LAN adapter and the partner's address is correctly defined.

Ensure that at least one client is connected to the SCADAs. A SCADA will not switch to Standby if there are no active client connections to its partner, since it cannot tell if the partner’s network is OK.

iClient with logical node names cannot connect to a partner SCADA.

Make sure that the iClient is configured properly. Refer to the Configuring iClients section.

Make sure the iClient machine can ping the SCADA node(s) using the iFIX node name.

Verify the Hosts files are configured correctly on all node links. For more information, refer to the Before You Begin with TCP/IP section in the Setting Up the Environment e-book.

The I/O driver configuration is causing issues. For instance, blocks are going off scan, or question marks appear for data links (the default indication that there is no data, and is defined in the User Preferences).

Your driver configuration or node configuration (where the driver is installed) could be invalid. Try running each node independently before configuring Enhanced Failover to determine if the drivers are configured differently on each node. They should be configured identically.

When the active switches to the standby node, the blocks go off scan and you see @@@@ signs or question marks instead of real data. The node fails back to the other node. All blocks are still off scan, after it fails back.

Your standby node is most likely not configured properly. Your driver configuration could be invalid. Try running each node independently before configuring Enhanced Failover to determine if the drivers are configured differently on each node. Make sure that your drivers are configured the same on each node. Validate that both systems run properly alone before reconfiguring Enhanced Failover.

iClient nodes display error message number 1914, every time the active SCADA switches.

This is an expected message if the 1914 error has not been configured to be filtered. When an iClient establishes a connection to an active SCADA Server node in run mode, the iClient starts to read data from that node. When the active SCADA Server node switches to the partner SCADA, the iClient momentarily loses its session with that node, causing this error to appear.

You can suppress this error from appearing on screen. For more information on how to suppress this message and others, refer to the Reading Data from iFIX Pictures in iClients section.

Connection Not Established With Node.

When any iClient loses its iFIX networking session with a remote node, this error to appears in alarm service destinations. Check your Ethernet connections.

iFIXNotification dialog displays for an extended length of time.

iFIXNotification dialog displays when iClient is only able to communicate with a standby SCADA node. Check your Ethernet connections being used for iFIX networking.

Standby SCADA displays “Connection Failover: failover attempted” message every minute in alarm services destinations.

This is not a synchronization message. This message indicates that the iFIX networking connection to the Active SCADA has failed. Check the Ethernet connection being used for iFIX networking.

Failover takes a long time.

Adjust the session timers. Refer to the Working with Configurable Session Timers section in the Setting Up the Environment manual for more information.

On iClients, alarms are being lost during failover periods.

Make sure that the Alarm Startup Queue is enabled and that the queue size is large enough to handle the alarms. Refer to the Implementing Alarms and Messages manual for more information on this service.

On iClients, alarms are missing from the summary link or alarms appear and disappear in the alarm summary link.

Make sure that the Alarm Summary Queue on the node is twice as big as it would normally be if the node were not part of a SCADA server failover pair. For example, if you are generating 500 alarms, the Alarm Summary Queue must be set to 1000. Refer to the Implementing Alarms and Messages manual for more information on queue sizes.

You are not seeing any alarms.

Use the Alarm Status Utility (almstat.exe) to check if alarms are coming into the alarm queues and being taken out. To start this utility while iFIX is running, type the following at the command line:

almstat

If this utility verifies there are no alarms coming in or out, then you need to troubleshoot your alarm configuration or cabling. For more information, refer to the Implementing Alarms and Messages manual.

A View node or iClient connects to a secondary SCADA. The secondary SCADA remains in standby mode, instead of connecting to the primary node. Alarms do not display, and datalinks are not updating on secondary, View, or iClient.

Check the network connections used with iFIX networking. A network cable could be unplugged

After invoking the Database Manager on an iClient (View node), the list of available nodes does not contain the names of the expected SCADA nodes.

Wait until the iFIX network session is established to a redundant SCADA for all the node names to display in the list.  

Client applications (such as Workspace) running local on the standby SCADA node do not connect to the active SCADA node.

Client applications running locally on the failover SCADA nodes are dependent on an operational iFIX networking path between the SCADA nodes.

Ensure that after a disruption of the synchronization path followed by a disruption of the iFIX network path that both network paths are once again operational.  

When only the synchronization path is restored and the iFIX network path between the SCADAs remains bad, the client applications on the local SCADA can only connect to the local SCADA.  In this situation, the client applications on the standby SCADA are not able to “write” to the SCADA node.”

To check the iFIX networking path between the SCADA nodes, run NetHis.EXE on each SCADA node. The partner SCADA node should be listed as an established connection.