Manual Recalculation

The Manual re-calc/re-replicate option is often the best choice for generating past derived data.
Note: If you perform a server-to-server recalculation on source and destination servers whose clocks are not synchronized, extra data points may appear and original data points may not be recalculated. To ensure this does not occur, ensure the time is synchronized on both source and destination servers.

S2S/S2C collector Backfill procedure

With the Recalculate feature you can recalculate all tags for the time period during and after the connection loss. The recalculated tags will use the most accurate values in calculations.

During the period of connection loss, the collector buffers the data. When the connection is restored, the buffered data is forwarded to the Historian Server. When the buffered data arrives, the timestamps show earlier time than the most recent calculation timestamp.

Since the timestamp is earlier, the polled calculations will not execute again with the new data but the unsolicited calculations will re-trigger. Therefore, it is possible that calculations performed for tags during and after the connection loss might be not be entirely accurate.

Run S2C Backfill via Command line
ihServerToServerCollector.exe RELOADFILENAME=[file location] 
RELOADUSERNAME=[Username] <start time> <end time>

Example: C:\Program Files\Proficy\Proficy Historian\x86\Server>ihServerToServerCollector.exe RELOADFILENAME=c:\taglist.txt RELOADUSERNAME=\Administrator 1516875659 1516875785

For multi-instance support, the command requires the interface name as shown in the following example:

Example for Multi-instance Support: C:\Program Files (x86)\GE Digital\Historian Server to Server Collector\Server>ihServerToServerCollector.exe RELOADFILENAME=c:\taglist.txt RELOADUSERNAME=\Administrator 1669462600 1669463200 REG=sekhartest05_To_PredixOFFSS

See the following information about the parameters:
  • RELOADFILENAME: This is an optional parameter. File name should be absolute path, this file consists of the tag names, for which Backfill should be performed, each tag should be separated by new line. Any discrepancies in the file/no file exists/parameter not provided leads to Backfill all the tags related to the collector at the current time. After the Backfill, file gets deleted.
  • RELOADUSERNAME: This is an optional parameter. This username is used only when destination server is Historian for auditing purpose, and gets ignored when the destination is cloud.
  • TIMESTAMP: This parameter accepts Start and end time in seconds in epoch format for which Backfill should happen. https://www.epochconverter.com/
How Data Recovery works:
  • When the recovery logic is executed, the collector will setup subscriptions for all the trigger tags.
  • Next, it will recover data. The collector first determines how long it has been since the last write. It compares the current time to data in the registry key LastCalcRepWriteTime, which stores the last time data was written to the archive. The collector compares this to the Max Recovery Time that is specified in the user settings and performs a raw data query on the shorter of these two periods. Then it will take the shorter of these two and do a raw data query for all trigger tags. It will then process the returned samples in sequential order based on time. For example, if the collector was shut down for 8 hours, but Max Recovery Time was 4 hours, only 4 hours of data would be recovered.
  • Recovery is performed before real time processing. Once recovery is complete, it will start polling and processing subscriptions in real time. The subscriptions in real time are queued up till the recovery is done.
  • Recovery logic will place an end-of-collection marker at the point in time where the collector was shut down. This end-of-collection marker may or may not be there once the recovery is complete. As part of recovery logic, if it calculates a data point exactly at that timestamp where the end-of-collection marker is there, then it will be overwritten with the calculated good data.
  • The recovery logic does not write samples to trigger tags or tags that are just in the formula. It is intended to write samples to the calculation tags.
  • Messages are added to the log file that indicate when entering and exiting recovery mode.

Examples

The examples below assume the following tag configuration.
  • Machine 1:

    Runs Data Archiver, iFIX collector (Collector 1), and Calculation collectors.

  • Machine 2:

    Runs iFIX collector (Collector 2), which collects and sends data to the archiver in Machine 1 (as a Remote Collector).

    TagA and TagB are the iFix tags coming from Collector1 and Collector2, respectively. Both of these tags are scanned at a 1-minute poll rate.

The following example demonstrates the recovery function for an unsolicited 1-minute interval calculation tag that has a simple current value function.

Create an event based 1-minute interval Calculation Tag (CalcTag1) in Machine 1 consisting of the following calculation: Result=CurrentValue (TagA)

Stop the Calculation collector for 5 minutes and then restart it to trigger data recovery for the 5-minute shutdown period. For the following example, the Calculation collector was stopped at 2002-12-27 17:05:36 and started at 2002-12-27 17:10:48.

Since there is no interruption for the iFIX collector, the raw data query for TagA results the following output:

Raw Data Query for TagA during shutdown period

114) 81 [2002-12-27 17:02:00:00000] Good NonSpecific

115) 72 [2002-12-27 17:03:00:00000] Good NonSpecific

116) 64 [2002-12-27 17:04:00:00000] Good NonSpecific

117) 56 [2002-12-27 17:05:00:00000] Good NonSpecific

118) 39 [2002-12-27 17:06:00:00000] Good NonSpecific

119) 31 [2002-12-27 17:07:00:00000] Good NonSpecific

120) 22 [2002-12-27 17:08:00:00000] Good NonSpecific

121) 14 [2002-12-27 17:09:00:00000] Good NonSpecific

122) 6 [2002-12-27 17:10:00:00000] Good NonSpecific

A raw data query for CalcTag1 during the shutdown period generates the following:

Raw Data Query for CalcTag1 (before recovery)

96) 81 [2002-12-27 17:02:00:00000] Good NonSpecific

97) 72 [2002-12-27 17:03:00:00000] Good NonSpecific

98) 64 [2002-12-27 17:04:00:00000] Good NonSpecific

99) 56 [2002-12-27 17:05:00:00000] Good NonSpecific

100) 0 [2002-12-27 17:05:36:00000] Bad OffLine

Note that an end-of-collection marker is placed at the shutdown point (that is, at 17:05:36) with a bad data quality.

Once the recovery is complete, this is what we see for the recovered CalcTag1. Note that data during the shutdown period is recovered completely. Compare this result set with the one for TagA. Both are the same.

Raw Data Query for CalcTag1 (after recovery)

96) 81 [2002-12-27 17:02:00:00000] Good NonSpecific

97) 72 [2002-12-27 17:03:00:00000] Good NonSpecific

98) 64 [2002-12-27 17:04:00:00000] Good NonSpecific

99) 56 [2002-12-27 17:05:00:00000] Good NonSpecific

100) 0 [2002-12-27 17:05:36:00000] Bad OffLine

101) 39 [2002-12-27 17:06:00:00000] Good NonSpecific

102) 31 [2002-12-27 17:07:00:00000] Good NonSpecific

103) 22 [2002-12-27 17:08:00:00000] Good NonSpecific

104) 14 [2002-12-27 17:09:00:00000] Good NonSpecific

105) 6 [2002-12-27 17:10:00:00000] Good NonSpecific

Also note that the end-of-collection marker is not overwritten by the recovery logic here. If it calculated a data point exactly at the end-of-collection marker, then it would have been overwritten by the calculated good value.

The following example demonstrates the recovery function for an unsolicited calculation tag that has multiple triggers.

Create an event based Calculation Tag (CalcTag2) in Machine 1 consisting of the following calculation:

Result=CurrentValue (TagA) + CurrentValue (TagB)

where TagA and TagB are both trigger tags, coming from Collector1 and Collector2 respectively. Set the collection offset of 5 seconds for TagA and 10 seconds for TagB, forcing the calculation to be performed twice per minute.

Stop the Calculation collector for 5 minutes, and then restart it to trigger data recovery for this 5-minutes shutdown period. For the following example, the Calculation collector was stopped at 02/18/2003 12:15:33 and started at 02/18/2003 12:21:53.

Since the iFIX collector was not interrupted, a raw data query for TagA and TagB values generates the following output:

Raw Data Query for TagA during the shutdown period

10) 13 [2003-02-18 12:10:05:00000] Good NonSpecific

11) 12 [2003-02-18 12:11:05:00000] Good NonSpecific

12) 11 [2003-02-18 12:12:05:00000] Good NonSpecific

13) 11 [2003-02-18 12:13:05:00000] Good NonSpecific

14) 10 [2003-02-18 12:14:05:00000] Good NonSpecific

15) 18 [2003-02-18 12:15:05:00000] Good NonSpecific

16) 17 [2003-02-18 12:16:05:00000] Good NonSpecific

17) 16 [2003-02-18 12:17:05:00000] Good NonSpecific

18) 16 [2003-02-18 12:18:05:00000] Good NonSpecific

19) 15 [2003-02-18 12:19:05:00000] Good NonSpecific

20) 14 [2003-02-18 12:20:05:00000] Good NonSpecific

21) 13 [2003-02-18 12:21:05:00000] Good NonSpecific

Raw Data Query for TagB during the shutdown period

10) 35 [2003-02-18 12:10:10:00000] Good NonSpecific

11) 34 [2003-02-18 12:11:10:00000] Good NonSpecific

12) 33 [2003-02-18 12:12:10:00000] Good NonSpecific

13) 32 [2003-02-18 12:13:10:00000] Good NonSpecific

14) 31 [2003-02-18 12:14:10:00000] Good NonSpecific

15) 31 [2003-02-18 12:15:10:00000] Good NonSpecific

16) 39 [2003-02-18 12:16:10:00000] Good NonSpecific

17) 38 [2003-02-18 12:17:10:00000] Good NonSpecific

18) 37 [2003-02-18 12:18:10:00000] Good NonSpecific

19) 36 [2003-02-18 12:19:10:00000] Good NonSpecific

20) 36 [2003-02-18 12:20:10:00000] Good NonSpecific

21) 35 [2003-02-18 12:21:10:00000] Good NonSpecific

A raw data query for CalcTag2 during the shutdown period generates the following:

Raw Data Query for CalcTag2 (before recovery)

12) 50 [2003-02-18 12:09:05:00000] Good NonSpecific

13) 50 [2003-02-18 12:09:10:00000] Good NonSpecific

14) 49 [2003-02-18 12:10:05:00000] Good NonSpecific

15) 48 [2003-02-18 12:10:10:00000] Good NonSpecific

16) 47 [2003-02-18 12:11:05:00000] Good NonSpecific

17) 46 [2003-02-18 12:11:10:00000] Good NonSpecific

18) 45 [2003-02-18 12:12:05:00000] Good NonSpecific

19) 44 [2003-02-18 12:12:10:00000] Good NonSpecific

20) 44 [2003-02-18 12:13:05:00000] Good NonSpecific

21) 43 [2003-02-18 12:13:10:00000] Good NonSpecific

22) 42 [2003-02-18 12:14:05:00000] Good NonSpecific

23) 41 [2003-02-18 12:14:10:00000] Good NonSpecific

24) 49 [2003-02-18 12:15:05:00000] Good NonSpecific

25) 49 [2003-02-18 12:15:10:00000] Good NonSpecific

26) 0 [2003-02-18 12:15:11:00000] Bad OffLine

Once data recovery is complete, this is what we see for the recovered data for CalcTag2. Note that data during the shutdown period is completely recovered:

Raw Data Query for CalcTag2 (after recovery)

12) 50 [2003-02-18 12:09:05:00000] Good NonSpecific

13) 50 [2003-02-18 12:09:10:00000] Good NonSpecific

14) 49 [2003-02-18 12:10:05:00000] Good NonSpecific

15) 48 [2003-02-18 12:10:10:00000] Good NonSpecific

16) 47 [2003-02-18 12:11:05:00000] Good NonSpecific

17) 46 [2003-02-18 12:11:10:00000] Good NonSpecific

18) 45 [2003-02-18 12:12:05:00000] Good NonSpecific

19) 44 [2003-02-18 12:12:10:00000] Good NonSpecific

20) 44 [2003-02-18 12:13:05:00000] Good NonSpecific

21) 43 [2003-02-18 12:13:10:00000] Good NonSpecific

22) 42 [2003-02-18 12:14:05:00000] Good NonSpecific

23) 41 [2003-02-18 12:14:10:00000] Good NonSpecific

24) 49 [2003-02-18 12:15:05:00000] Good NonSpecific

25) 49 [2003-02-18 12:15:10:00000] Good NonSpecific

26) 0 [2003-02-18 12:15:11:00000] Bad OffLine

27) 48 [2003-02-18 12:16:05:00000] Good NonSpecific

28) 56 [2003-02-18 12:16:10:00000] Good NonSpecific

29) 55 [2003-02-18 12:17:05:00000] Good NonSpecific

30) 54 [2003-02-18 12:17:10:00000] Good NonSpecific

31) 54 [2003-02-18 12:18:05:00000] Good NonSpecific

32) 53 [2003-02-18 12:18:10:00000] Good NonSpecific

33) 52 [2003-02-18 12:19:05:00000] Good NonSpecific

34) 51 [2003-02-18 12:19:10:00000] Good NonSpecific

35) 50 [2003-02-18 12:20:05:00000] Good NonSpecific

36) 50 [2003-02-18 12:20:10:00000] Good NonSpecific

37) 49 [2003-02-18 12:21:05:00000] Good NonSpecific

38) 48 [2003-02-18 12:21:10:00000] Good NonSpecific

39) 47 [2003-02-18 12:22:05:00000] Good NonSpecific

40) 46 [2003-02-18 12:22:10:00000] Good NonSpecific

The following example demonstrates the recovery function for an unsolicited calculation tag that has multiple triggers, but for which none of the triggers is in the formula.

TagA and TagB are the iFix tags coming from Collector1 and Collector2, respectively. Both tags are scanned at a 1-minute poll rate. This example uses two more iFix tags, TagC and TagD, coming from Collector1.

Create an event-based Calculation Tag (CalcTag3) in Machine 1 consisting of the following calculation:

Result=CurrentValue (TagA) + CurrentValue (TagB)

Make sure that the trigger tags for this calculation tag are TagC and TagD, which are not in the formula. Set the collection offset of 5 seconds for TagC and 10 seconds for TagD, forcing the calculation to be performed twice per minute.

Stop the Calculation collector for 5 minutes, and then restart it to trigger data recovery for this 5-minutes shutdown period. For the following example, the Calculation collector was stopped at 02/18/2003 02:24:37 and started at 02/18/2003 02:31:44.

Since the iFIX collector was not interrupted, a raw data query for TagA and TagB values generates the following output:

Raw Data Query for TagA during shutdown period

56) 13 [2003-02-18 14:21:05:00000] Good NonSpecific

57) 12 [2003-02-18 14:22:05:00000] Good NonSpecific

58) 11 [2003-02-18 14:23:05:00000] Good NonSpecific

59) 11 [2003-02-18 14:24:05:00000] Good NonSpecific

60) 10 [2003-02-18 14:25:05:00000] Good NonSpecific

61) 19 [2003-02-18 14:26:05:00000] Good NonSpecific

62) 18 [2003-02-18 14:27:05:00000] Good NonSpecific

63) 17 [2003-02-18 14:28:05:00000] Good NonSpecific

64) 16 [2003-02-18 14:29:05:00000] Good NonSpecific

65) 16 [2003-02-18 14:30:05:00000] Good NonSpecific

66) 15 [2003-02-18 14:31:05:00000] Good NonSpecific

Raw Data Query for TagB during shutdown period

141) 36 [2003-02-18 14:20:10:00000] Good NonSpecific

142) 36 [2003-02-18 14:21:10:00000] Good NonSpecific

143) 35 [2003-02-18 14:22:10:00000] Good NonSpecific

144) 34 [2003-02-18 14:23:10:00000] Good NonSpecific

145) 33 [2003-02-18 14:24:10:00000] Good NonSpecific

146) 32 [2003-02-18 14:25:10:00000] Good NonSpecific

147) 31 [2003-02-18 14:26:10:00000] Good NonSpecific

148) 31 [2003-02-18 14:27:10:00000] Good NonSpecific

149) 39 [2003-02-18 14:28:10:00000] Good NonSpecific

150) 38 [2003-02-18 14:29:10:00000] Good NonSpecific

151) 37 [2003-02-18 14:30:10:00000] Good NonSpecific

152) 36 [2003-02-18 14:31:10:00000] Good NonSpecific

A raw data query for CalcTag3 during the shutdown period generates the following:

Raw Data Query for CalcTag3 (before recovery)

6) 49 [2003-02-18 14:21:05:00000] Good NonSpecific

7) 49 [2003-02-18 14:21:10:00000] Good NonSpecific

8) 48 [2003-02-18 14:22:05:00000] Good NonSpecific

9) 47 [2003-02-18 14:22:10:00000] Good NonSpecific

10) 46 [2003-02-18 14:23:05:00000] Good NonSpecific

11) 45 [2003-02-18 14:23:10:00000] Good NonSpecific

12) 45 [2003-02-18 14:24:05:00000] Good NonSpecific

13) 44 [2003-02-18 14:24:10:00000] Good NonSpecific

14) 0 [2003-02-18 14:24:11:00000] Bad OffLine

A data query for the recovered CalcTag3 values once data recovery is complete generates the following. Note that data during the shutdown period is completely recovered:

Raw Data Query for CalcTag3 (after recovery)

6) 49 [2003-02-18 14:21:05:00000] Good NonSpecific

7) 49 [2003-02-18 14:21:10:00000] Good NonSpecific

8) 48 [2003-02-18 14:22:05:00000] Good NonSpecific

9) 47 [2003-02-18 14:22:10:00000] Good NonSpecific

10) 46 [2003-02-18 14:23:05:00000] Good NonSpecific

11) 45 [2003-02-18 14:23:10:00000] Good NonSpecific

12) 45 [2003-02-18 14:24:05:00000] Good NonSpecific

13) 44 [2003-02-18 14:24:10:00000] Good NonSpecific

14) 0 [2003-02-18 14:24:11:00000] Bad OffLine

15) 43 [2003-02-18 14:25:05:00000] Good NonSpecific

16) 42 [2003-02-18 14:25:10:00000] Good NonSpecific

17) 51 [2003-02-18 14:26:05:00000] Good NonSpecific

18) 50 [2003-02-18 14:26:10:00000] Good NonSpecific

19) 49 [2003-02-18 14:27:05:00000] Good NonSpecific

20) 49 [2003-02-18 14:27:10:00000] Good NonSpecific

21) 48 [2003-02-18 14:28:05:00000] Good NonSpecific

22) 56 [2003-02-18 14:28:10:00000] Good NonSpecific

23) 55 [2003-02-18 14:29:05:00000] Good NonSpecific

24) 54 [2003-02-18 14:29:10:00000] Good NonSpecific

25) 54 [2003-02-18 14:30:05:00000] Good NonSpecific

26) 53 [2003-02-18 14:30:10:00000] Good NonSpecific

27) 52 [2003-02-18 14:31:05:00000] Good NonSpecific

28) 51 [2003-02-18 14:31:10:00000] Good NonSpecific

29) 49 [2003-02-18 14:32:05:00000] Good NonSpecific

30) 49 [2003-02-18 14:32:10:00000] Good NonSpecific

31) 48 [2003-02-18 14:33:05:00000] Good NonSpecific

32) 47 [2003-02-18 14:33:10:00000] Good NonSpecific