Troubleshooting Checklist for Enhanced Failover Configuration

If you experience issues with Enhanced Failover and you do not know where to begin, start by asking yourself:

  1. Do you understand the architecture of your Enhanced Failover system? If not, review the What is Enhanced Failover?, What is my SCADA Role?, and Components of Enhanced Failover sections, for starters.  Other information is also provided earlier in this e-book.
  2. Once you understand the architecture, verify that each SCADA node works independently without Enhanced Failover enabled. For tips, refer to the Testing Your Enhanced Failover Configuration section.
  3. Confirm that you configured the primary and secondary nodes appropriately. Both SCADA computers in the pair should be configured the same way. For a helpful checklist on what to check, refer to the Preparation Checklist for Enhanced Failover section. Be sure to investigate each and every item in the list. If one SCADA has a different configuration, say the drivers and polling times are different, or the Database Dynamos (loadable blocks) are not loaded in the same slots, or network (NIC) cards are configured slightly differently, it will impact your system.
  4. Ensure that the iFIX Enhanced Failover option is enabled on both the primary and secondary SCADAs, that both SCADAs are using the same protocol (TCP or UDP), and that your client nodes are configured properly. For more information, refer to the Checklist for Enhanced Failover Configuration section.
  5. Check the Limitations of Enhanced Failover section. Is your configuration in compliance?
  6. If after all those steps, you believe everything is configured properly, the next step is to investigate what is occurring between the network cards. For instance, are your network cards doing what they are configured to do? In the Windows Task Manager, use the Networking tab to investigate if each adapter is performing appropriately. From the Networking tab, look for:
    • 1-Gigabit-Ethernet cards (or better) dedicated for SCADA-to-SCADA traffic (for a total of two network cards, minimum). For UDP, the dedicated SCADA-to-SCADA network cards should be excluded from the iFIX-to-iFIX network (not enabled for LAN redundancy). If either card is less than 1 Gigabyte-Ethernet, it's not a supported configuration. Also, both network cards must be of the same speed.
    • Does each network card have an Adapter name?  Are both cards of the same make and model number, and use the same drivers?
    • What is the network utilization of each card and what is the state of each card? Is it operational? For UDP only, is Jumbo Frames technology used on the dedicated network for Enhanced Failover? (Jumbo Frames technology allows for an Ethernet frame of 9000 MTU for the payload, compared to a frame of 1500 bytes without the Jumbo Frames.) Jumbo Frames technology must be used on the dedicated network for Enhanced Failover.
    • Be aware that network card configuration is one of the top configuration issues. For additional details, refer to the More Information on Network Card Usage section below.
  1. If after all of the network configuration checks, you believe everything is configured properly, the next step is to investigate the networking hardware.
    • Are all network cables and connections properly connected? Are you using a direct connection via a Cat6 crossover cable, without going through any switches, hubs, or routers (for UDP only)?
    • Do both Gigabit-Ethernet (or better) cards dedicated for SCADA-to-SCADA traffic have the same make and model number imprinted on the actual card or recorded in the user manual (for UDP only)?
  1. Finally, another place to investigate is in the Windows Task Manager, on the Performance tab. What is the physical memory available and system cache? Is your computer performing at the necessary level for Enhanced Failover configurations?  Be aware that:
    • The wake-up upon receiving a socket or/and request feature must be disabled on the SCADA-to-SCADA network card.
    • SpeedStep® technology is not supported and must not be enabled.
    • Due to limited bandwidth and latency, it is strongly advised that you do not use wireless networking technology in your networking solution.
    • The power save settings on your computers and dedicated network card (NIC) must be disabled. Do not use any power setting features that affect CPU clock speed.

More Information on Network Card Usage

Network cards (NICs) may be used for a multitude of tasks within iFIX. The following tasks may each use a separate NIC:

  • Driver I/O to PLC connectivity
  • SCADA synchronization
  • iFIX Client connectivity
  • Non-iFIX activity such as company network access

It is important to understand the use of all the network cards on the PC. It is possible to use a particular NIC card for multiple uses by design. It is also possible to inadvertently apply the NIC card to the incorrect uses causing undesirable results.

Tips for working with NICs

  • Whenever a new NIC is installed, re-check all configuration items that use the NIC card. The NIC slot/order or IP address could require changes.
  • It is desirable that both machines have the same NIC cards and NIC order (slot).
  • Some drivers use a NIC IP Address or slot/order number. Communication may cease after installing another NIC card since slot/order may change. The slots should be in the same order on both nodes.
  • When copying a Driver Configuration file from one node to another, the slot/order or IP Address may need to be changed on the PC receiving the file.
  • It is required to perform SCADA synchronization on a dedicated NIC (for UDP only). It is recommended to have an identical NIC on both the Primary and Secondary SCADA.
  • SCADA synchronization should be tested with the “default” NIC settings. Any advanced setting changes should be mirrored on the partner SCADA.
  • SCADA synchronization can optionally include a second NIC (as a backup to the first). Often this can be the same NIC as the iFIX Client connection.
  • Typically, iFIX Client connectivity uses a different NIC then SCADA synchronization. Only ONE NIC should be enabled in the iFIX Networking configuration (unless LAN Redundancy is used).

See Also