Troubleshooting

Solve Minor Operating Problems

The following is a table of troubleshooting tips for solving minor operating problems with Historian.
Issue Suggested Action
After setting the system clock back, browsing the collector from Historian Administrator produces a Visual Basic script error. Delete temporary Internet files and restart Internet Explorer.
With the Historian Non-Web Administrator, switching usernames causes the system to reject the login if the User must change password at next login option is selected at time of user creation. New users with this setting must log in to the appropriate Windows operating system at least once. If the login is rejected, drag the Non-Web Administrator to your desktop, right-select the icon, select "Run as . . .", enter a new username and password, and select OK. The login of the new user is then accepted.
A long error message from the File collector does not fit within the message field in the Historian File collector console window, resulting a partial display of the message. Open the File collector log file to see the complete text of the error message.
Excel Sample Reports do not display data. When opening a Sample Excel report, you may receive a message prompting you to update all linked information in the workbook (Yes) or keep the existing information (No). It is recommended that you select No and keep the existing information. The links will be automatically updated for your worksheet. Save your worksheet after the links have been updated.
Need to connect an Historian Server to an Historian Client through a firewall. Open port 14000 to enable client to server connection through a firewall.
Receiving archive offline failed writes messages. These occur when the timestamps of data being sent to the archiver are not in the valid time range of any online archives. For example, failed writes occur when the timestamps appear before the oldest archive, the archive is offline, or timestamps are more than 15 minutes past the current time on another archiver.

FAQ: Run a Collector as a Service

The following list is frequently asked questions about running a collector as a service.

Can all collectors be run as a Windows service? If not, which ones cannot?
The OPC Collector, File collector, Simulation Collector, Calculation collector, and Server-to- Server Collector can be run as services. The iFIX collector run as a background task and cannot be run as a service.
Can all collectors be run as an application? If not, which ones cannot?
All collectors can run as applications (console programs). This includes the Simulation Collector. To make a collector run as a console program, pass a RUNASDOS command line parameter.
What does "running as a service" mean?

It means that the collector appears in the Control Panel list of services. It can run at system boot or be run with a different username and password from the currently logged-in user.

How can the iFIX collector be set up to run when no one is logged in?
It can be set up to run without a user login by adding it to the iFIX SCU task list as a background task and by configuring iFIX to continue running after logoff in local startup.
How do you shut down a collector running as console application?

Collectors started as console applications should be shut down by typing S at the command prompt in the DOS window and pressing Enter.

Can a collector be run as a Windows service and then stopped and restarted?

Yes. Collectors that can run as a service can be stopped and started in Control Panel Services. They can be paused/resumed through Historian Administrator.

What is the difference between running a collector to start as a service on boot up using the Services applet in Control Panel versus running iFIX as a service, which starts the collector through the startup task configuration in the SCU?

The collectors that can run as a service would not be started from iFIX. They can be started from the Control Panel start at system boot. Although you cannot run an iFIX collector as a service, you can log off and on while it is running.

Changing the Base Name of Automatically Created Archives

About this task

When the IHC file is created, it stores the name of the server inside the IHC file. Automatically created archives use that server name from the IHC file as a base name, not the Base Archive Name configured in Historian Administrator.

When you manually create an archive, however, the archive uses the Base Archive Name from Historian Administrator.

If you move an IHC file from one machine to another, you may want to change the default base archive name to match the new server. To change the default Base Archive Name, create a new .IHC file.

Procedure

  1. Export your tags using the Excel Add-in.
    The Fields to Export window appears.
  2. Select all tag attributes and select OK.
    The data is exported to a new Excel worksheet.
  3. Examine the Comments column in the new worksheet. To ensure a clean import, the Comments column must be completely full or completely empty. If no comments are found, this column must be deleted to ensure a clean import. If only some comments are missing, the missing fields must be filled out with comments.
  4. Save the Excel spreadsheet.
  5. Stop the Historian Data Archiver service.
  6. Open the default archive path in Windows Explorer and rename the .IHC file.
    Rename MyMachine.IHC to MyMachine.OLD.
  7. Restart the Historian Data Archiver service.
    A new, blank IHC file is created for the machine.
  8. Import your tags to this new configuration using the Excel Add-In.

Configuring the Inactive Timeout Value

About this task

The Non-Web Administrator offers a configurable timeout. This configurable timeout determines how long the Non-Web Administrator will wait before severing its connection to an inactive Historian archive. The default timeout value is 90 seconds.

Procedure

  1. Assuming Historian is already open, double-click on the Main button to open Historian Administrator Login window.
  2. Select the Browse for Server button.
  3. Select the Historian server you wish to configure from the Servers list.
  4. In the Connection Timeout field, select the Use Value option and enter a timeout value in seconds.

Configuring Deep Data Tree Warnings

About this task

Reading and writing to deep trees with large time ranges can be very inefficient. Create a MaxIndexRecursionDepth registry key to configure the depth at which the archiver will warn about deep data trees.

Procedure

  1. From the Start menu, select Run, and then enter Regedit.
  2. Open the following key folder: HKEY_LOCAL_MACHINE\SOFTWARE\Intellution, Inc.\iHistorian\Services\DataArchiver
  3. Create a value as DWORD called MaxIndexRecursionDepth.
  4. Set MaxIndexRecursionDepth to a number higher or lower than the default value of 900.
    To get more warning messages, set a smaller number if you have an archive which is 10 to 100 deep.
  5. Select OK.
  6. Close the Registry Editor.
  7. Restart the archiver for the changes to take effect.

Control Data Flow Speeds with Registry Keys

Configure buffer flush speed with the BufferFlushMultiplier key

Store-and-forward buffering is a key feature of Historian collectors. It prevents data loss during planned or unplanned network outages between a collector and Historian server.

If the collector is disconnected from the archiver for several hours or days, many megabytes of data can be buffered and must be delivered by the collector to the Data Archiver upon reconnect. Since all data is sent from the collector to the archiver in time order, the design goal has been to catch up to real time as quickly as possible by sending data as fast as possible.

If this is not the desired behavior because you want to limit the network load on a slow, shared Wide Area Network (WAN) or you want to limit the CPU load on the Data Archiver caused by the incoming data, you can configure the collector to throttle the data it is sending. Throttling Store and Forward does not affect the Alarms and Events collector.

Important: Because data is sent in time order (oldest first), you will not be able to retrieve current historical data until the throttled flush is complete.

Configuring the throttle is easy, but it requires modifying a registry key, so it should be done with caution.

A DWORD registry key called BufferFlushMultiplier is present under each collector. For Windows 32-bit, it is located under HKey_Local_Machine\Software\Intellution, Inc.\iHistorian\Services\YOUR_COLLECTOR_TYPE. For Windows 64-bit, it is located under HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Intellution, Inc.\iHistorian\Services\YOUR_COLLECTOR_TYPE.

  • To slow the store and forward throttling, set the value of BufferFlushMultiplier to 2. The 2 means that the collector should never send data at more than 2 times its normal rate to limit network and CPU load.
  • To disable throttling, set the value of BufferFlushMultiplier to 0 or delete the registry key.

Control archiver speed with the NumIntervalsFlush registry key

The NumIntervalsFlush registry key controls how quickly the collector sends data to the archiver. The collector collects from the data source at the user configured rate, but for efficiency it bundles data samples in a single write to archive. By default, the collector will send data to archiver every 2 seconds or 10,000 samples, whichever happens first. Most often, it sends every 2 seconds because the collector is not collecting that many samples that fast.

If you need collected data sent to archiver right away, so that it is available for retrieval or for calculations, use the NumIntervalsFlush registry key.

You will have to create the registry key, as it does not exist by default. Create a DWORD value called NumIntervalsFlush under the collector, in the same place as HISTORIANNODENAME and INTERFACENAME. On a 64-bit Windows Operating System, all 32-bit component-related registry keys (such as collectors, Client Tools, and APIs) will be located under HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Intellution, Inc.\iHistorian\.

The preferred setting for Num Intervals Before Flush is 5. The intervals are 100 millisecond increments. The default of 20 means (20 * 100msec) = 2000 msec = 2 seconds. Set the value to 5 and the collector will send every 500msec.
Note: Changes to the registry key do not take effect until the collector is restarted. This setting affects the sending of data whether it was collected polled or unsolicited.

Configure Inactive Server Reset Timeout

You can configure inactive server connections to reset automatically with the SocketRecvTimeOut registry key. SocketRecvTimeOut configures a timeout that forces the connection to drop and re-establish if no data is received during the specified time. Consider this configuration when your collector goes to status Unknown for long periods of time even when the connection between collector and archiver is good.

Create a DWORD registry key SocketRecvTimeOut under the collector where the problem is occurring and set to a value greater than 90 seconds. A typical value would be 300 seconds. If no bytes are received by the collector for 300 seconds, then the network connection will be closed and re-established.

Historian Errors and Message Codes

When you review errors and messages, for example, in an Historian archiver log file, full descriptions are usually included. If a number appears instead of a description, use the following table to determine the meaning of the error or message.
Table 1. Historian Error Codes and Messages
Number Description
-32 Operation not permitted
-31 The requested data store was not found
-30 The requested Enumerated Set was not found
-29 A supplied argument is outside the valid range
-28 A supplied argument is NULL
-27 A supplied argument is invalid
-26 Alarms and Events subsystem unreachable
-25 Attempted data delete outside allowed modification interval
-24 Data Retrieval Count Exceeded
-23 Invalid Server Version
-22 Backup Exceeded Space
-21 Calculation Circular Reference
-20 Not Licensed
-19 Duplicate Interface
-18 No Value
-17 License: Invalid License DLL
-16 License: Too Many Users
-15 License: Too Many Tags
-14 Invalid Tagname
-13 Write No Archive Available
-12 Write Outside Active
-11 Archive Read Only
-10 Write Archive Offline
-9 Write in Future
-8 Access Denied
-7 Not Valid User
-6 Duplicate Data
-5 Not Supported
-4 Interface Not Found
-3 Not Connected
-2 API Timeout
-1 FAILED
0 OK
0 Undefined
1 Connection Successful
2 Connection Unsuccessful
3 Audited Write
4 Audited Write Update
5 Audited Write Out Of Order
6 Audited Write Update Out Of Order
7 Message On Update
8 Message On Update Out Of Order
9 License Library Function Missing
10 License Library Missing
11 Failed Write
12 Tag Added
13 Tag Modified
14 Tag Deleted
15 Interface Added
16 Interface Modified
17 Interface Deleted
18 Archive Added
19 Archive Add Failure Time Overlap
20 Archive Deleted
21 Archive Overwritten
22 Archive Backup Began
23 Archive Backup Failed
24 Archive Backup Completed
25 Archive Five Days Till Closing
26 Archive Three Days Till Closing
27 Archive One Day Till Closing
28 License Key Removed
29 License Max Tags Exceeded
30 License Max Users Exceeded
31 License Max Tags Exceeded Shutdown
32 License Max Users Exceeded Shutdown
33 License Library Invalid
34 Buffer Normal
35 Buffer On Disk
36 Buffer Out Of Space
37 Incomplete Shutdown
38Archive Modified
39 License Expired
40 Buffer Could Not Create
41 Archiver Startup
42 Archiver Shutdown
43 Audit Status Changed
44 Option Modified
45 Write Processing Stopped
46 Write Processing Resumed
47 Interface Status Unknown
48 Archive Closed
49 Interface Stopped
50 Interface Started

Determining the Version of the Historian Server

About this task

The version of the Historian Server that Historian Administrator is connected to appears in the About window. If the version number is "Unknown", you are connected to an Historian v1.0 Server.

Procedure

  1. Open Historian Administrator.
  2. Select the About link.
    The About Historian window appears.
  3. Note the version number that appear in Versions section of the About Historian window, as shown in the following figure.


Using VBA to Return a List of Valid Field Options

About this task

Use VBA to determine which fields are available for an SDK object.

Procedure

  1. Open Microsoft Visual Basic for Applications (VBA).
  2. Ensure that you have selected the Historian Software Kit reference in the References window.
  3. Select the Object Browser from the View menu.
  4. Select Historian_SDK from the Library drop-down list.
  5. Select an item in the Classes list to generate a list of valid fields for that item.
    when you select ihCollectionType, you see a list of fields that includes Polled and Unsolicited.

Scheduled Software Performance Impact

Running continuous backup or disk scan software applications such as anti-virus scans, automated backup software, or any other software that accesses disk drives to a high degree may affect the overall performance of your Historian System by competing with Historian for disk resources.

If your Historian System requires that you need an extremely high throughput (20k/sec or greater), consider disabling the scheduled software execution.

Intellution 7.x Drivers as OPC Servers

The ABR and the ABC drivers are OPC v2.0 compliant. All other Intellution 7.x drivers, including the MB1, support OPC v1.0 compliance.

Version 7.x drivers also comply with the OLE for Process Control (OPC) v1.0a standard. Any 1.0-compliant OPC client application can access process hardware data through the I/O Server.

The ABC I/O Server also supports the v2.0 standard and with the OPC Alarms and Events v1.0 specification. Refer to Using OLE for Process Control (OPC) Functionality to learn more about the advantages of OPC.

Troubleshooting Failed Logins

The following is a table of error messages, possible causes, and recommended corrective actions for failed logins sometimes experienced with Historian Administrator.

Error MessageSuggested Action
User does not have authority to read messages. The user is NOT a member of iH Readers nor a member of the iH Security Admins security groups. To access the Main Page, the user must have read access.
[07/18/2001 03:00:46.071 PM] USER: DOMAIN1\administrator TOPIC: Security MSG: DOMAIN1\administrator(administrator) unsuccessfully connected at 07/18/2001 03:00:46.071 PM. Not able to establish session to the server from a remote Web-based Clients. Page cannot be displayed. Error in Internet Explorer. Check network connection or IIS on the server.
[07/17/2001 07:56:06.950 PM] USER: DataArchiver TOPIC: Security MSG: DataArchiver(DataArchiver) Exceeded number of licensed users at 07/17/2001 07:56:06.950 PM (NumUsers=0 MaxUsers=0)Outdated or failed HASP key is attached. Obtain new key from technical support.
[07/17/2001 07:58:18.980 PM] USER: \baduser TOPIC: Security MSG: \baduser(baduser) unsuccessfully connected at 07/17/2001 07:58:18.980 PM. Bad password for user account. Enter correct password.
[07/17/2001 07:58:48.712 PM] USER: \administrator TOPIC: Security MSG: \administrator(administrator) unsuccessfully connected at 07/17/2001 07:58:48.712 PM. DataArchiver service is not running. Results in a Failed to connect to server error. Make sure data archiver service is running and that the user did not enter a bad password.
[07/17/2001 07:23:44.397 PM] USER: DataArchiver TOPIC: Security MSG: DataArchiver(DataArchiver) Exceeded number of licensed tags at 07/17/2001 07:23:44.397 PM (NumTags=1021 MaxTags=0) Must Shutdown Number of configured tags exceeds number of tags allowed by the key. Delete enough tags to meet the license limit or obtain a license for more tags from technical support.
[07/17/2001 07:35:32.134 PM] USER: DataArchiver TOPIC: Security MSG: DataArchiver(DataArchiver) Licensed expired. Must shutdown 07/17/2001 07:35:32.134 PM. To troubleshoot, refer to the DataArchiver.log file. License on key has expired (archiver will not start). Obtain a new license from technical support.

Troubleshoot Data Collector Configuration

Troubleshooting Data Collector configuration and/or performance requires a thorough understanding of how Historian works and how the various parameters affect system operation. Armed with this knowledge, you can usually localize a problem to one or more functions or parameter settings and take effective corrective action.

Historian offers several tools to help find the cause of an operating problem.

LOG files
The system creates a new log file each time an archiver or collector is started. You can open these files in Notepad or another text editor. The -nn suffix in the file name indicates the place of each log file in the time sequence.
  • The data collector log files, located in the LogFiles folder within the Historian program folder, are a historical journal of every event affecting operation of the collector.
  • The DataArchiver-nn.LOG files are sequential files for the archiver only.
  • The iFIX collector-nn.LOG files contain performance information on iFIX collector functioning.
SHW file

The Data Collector .SHW file shows configuration data for collectors and is also located in the LogFiles directory under Historian. Verify that the parameter and configuration settings match what you configured. You can open this file in Notepad or another text editor.

Collector Maintenance Performance Indicators

These performance and status indicators can be a major aid in identifying, localizing, and diagnosing a problem with a collector.

Collector Maintenance pages

The Collector Maintenance pages can provide useful information about settings and selections of various options and parameter values. Examine each field and verify that it is appropriate for the current application.

iHistorianSDKerrors.log

When performing any troubleshooting of the File collector Administrator or SDK program you should examine the iHistorianSDKerrors.log file, which is located in the LogFiles folder in your Historian program folder.

Troubleshoot Tags

Tag Configuration

To diagnose a problem in tag configuration, examine the .LOG and .SHW files in the Historian/Logfiles directory. Since these files are a journal record of all system events and parameter modifications important to a system administrator, they can be helpful in identifying and localizing the source of a system malfunction or data error.

You may also find the Windows Event Viewer helpful in troubleshooting. For more information on the Windows Event Viewer, refer to Review System Alerts and Messages.

When troubleshooting SDK applications, such as Historian Administrator, the Excel Add-In, or File collector, examine the iHistorianSDKerrors.log file, located in the LogFiles folder in the Historian program folder.

Stale Tags

If you are not seeing all stale tags in the Historian Web Admin, the ClientManager service may be down. If so, the thread that processes stale tags is suspended until connection is restored.

If the connection to the ClientManager service is lost while processing a batch of 100 tags, that set of tags is skipped and a Tag Property Update error is logged. When the ClientManager service is back up again, the next set of 100 tags is considered for scanning. The skipped tags must wait until the Diagnostic Manager???s next cycle. Until then, they are not marked as stale or not stale.

Alternately, the DataArchiver or ConfigManager services may be down. If so, all the remaining tags are skipped, and a DataOpenRecordset/Tag Update failure is logged. Even though the thread that processes stale tags is not suspended, all the remaining tags have to wait until the Diagnostic Manager???s next cycle. Until then, they are not marked as stale or not stale.

Troubleshoot Historian Performance

There are many parameters that affect Historian Server performance and scalability. This topic emphasizes the parameters that help improve performance and scalability.

Buffer Memory Max

Buffer Memory Max allows you to specify the maximum memory buffer size that an archiver queue can take. Use this parameter to control memory allocation to the write queues. If you plan to collect large amounts of data with high update rates, increase the Buffer Memory Max size. Increasing the Buffer Memory Max size provides enough memory for write queues, thus increasing the Historian performance. Refer to The Global Options Section for more information.

Performance Counters

Historian provides a variety of performance counters that can be used to monitor how well the application is performing. Using the following performance counters you can determine the Historian Server performance and scalability:
  • Perftag_GenericWriteQueueSize: displays the total number of messages present in the Generic Write queue.
  • Perftag_FastReadQueueSize: displays the total number of messages present in the Fast Read queue.
  • Perftag_CollectorDataWriteQueueSize: displays the total number of messages present in the Collector Data Write queue.
  • Perftag_EventMonitorQueueSize: displays the total number of events present in the Event Monitor queue.
  • Perftag_DataWriteQueueSize: displays the total number of messages present in the Data Write queue
  • Perftag_ReadQueueSize: displays the total number of messages present in the Read queue
By monitoring the above performance counters, make sure that you manage application performance. Ideally, all write queues should be zero for optimum archiver performance. If you see an increase in back logs in write queues, then you can:
  • Control archive transitions by having larger archive file size.
  • Control update rate.
  • Perform a backup when an archiver is in stable state.
  • Control the caching evaluation period. Refer to Historian Performance Counters for more information.

System File Cache Tuning

Historian allows you to specify the maximum disk cache memory that an archiver can use. Ideally, Historian consumes 25% of system memory. If your computer has extra memory, then you can increase the disk cache memory to optimize Historian performance.
Note: You should not increase disk cache memory if you do not have the necessary system resources.

Troubleshoot the Archive Service

Archiver cannot create a new archive: If an archive is full and you have disabled Overwrite Old Archives in the Global Options section, Historian can only create a new archive if there is enough disk space available. If you do not have enough disk space to create a new archive, Historian does not generate an alert on the Main page, but instead adds the following messages to the Data Archiver log file: Could not locate empty archive file, nor create one, nor use old one. Stopped processing data write requests. Free up disk space or enable Overwrite Old Archives to continue archiving your data.

Archive service not starting automatically: On some systems the Archiver does not start automatically after install. Start the Archiver manually through the Services window or restart the computer. The Archiver should start on all future restarts. The Archiver service may also fail to start if it is started by an improperly configured domain account. In order to add a user from the domain as the Archiver service start up account, the machine Historian is installed on must have joined the domain. The user changing the service log on account must also have administrative rights to the Historian machine. This user the service is configured with must be a member of the Domain Administrators group and have administrative rights to the Historian machine. The user also needs to be allowed to log on as a service and act as part of the operating system. This is configured in the local policy editor of the Historian machine.

Investigate failed backups: Refer to the ArchiveBackup-XX.log file to investigate failed backups. The backup can fail if the user does not have valid permissions or if the archive name supplied is not valid. For more information, refer to Knowledge Base article: i014491 at http://iglobalcare.gefanucautomation.com.