Retrieving Data from Historian
About Retrieving Data from Historian
After data collection, the Historian Server compresses and stores the information in a Data Archive or a *.iha file. Any client application can retrieve archived data through the Historian API. The Historian API is a client/server programming interface that maintains connectivity to the Historian Server and provides functions for data storage and retrieval in a distributed network environment.
- Historian Analysis
- Knowledge Center
- iFIX
- CIMPLICITY
- Real-Time Information Portal
- Dream Reports
- Excel Add-In
- Custom SDK Applications
- OLE DB
Historian exposes various sampling and calculation modes that are used on retrieval of data that has already been collected to the archive. These modes do not effect data collection. Some sampling modes are suited to compressed data and should be used when collector compression or archive compression is used.
Sampling Modes
Sampling modes are used to specify how the data will be retrieved from Historian. Several modes are available, such as CurrentValue, Interpolated, Calculated and RawByTime. Sampling modes are specified in the client you use to retrieve data from Historian.
For more information, refer to the Advanced Topics section in the online help.
Sampling Mode | Results |
---|---|
CurrentValue | Retrieves the most recent data sample value received by the archiver, of any data quality. This does not necessarily produce the most recent raw data sample, as archive compression may not have stored the most recent raw sample. The sample has a time stamp, a value, and a quality. The time stamp returned is not typically the current time; it is the time stamp as sent by the collector. If you have a slow poll rate or if collector compression is enabled, the time stamp may be much earlier than the current time. |
RawByTime | Returns all raw samples of all qualities with a time stamp greater than a specified start time, and less than or equal to a specified end time. The RawByTime sampling mode will not return a sample equal to the start time. |
RawByNumber | Returns a specified number of samples of all qualities with a time stamp greater than or equal to the start time. The RawByNumber sampling mode will return a sample with a time stamp equal to the start time if one exists You must also specify a direction and number of samples when using this sampling mode. |
Interpolated | When archive or collector compression is used, a minimal number of actual data samples are stored in the archive. When this data is retrieved, interpolation can be performed to create an evenly spaced list of most likely real-world values (since the actual values have been removed from the archive through the compression algorithm). The Interpolated sampling mode is also useful for data samples that haven't undergone archive compression. For example, you may want to plot data along an interval that doesn't match the collected raw samples. Using the Interpolated sampling mode would give you the most likely real-world values for the specified period. Typically, you use the interpolated sampling mode when data is not collected on a set time schedule, or if you want to see the results returned in an interval that is slower than the collection rate. For example, these instances show when you can use interpolated mode to make evenly spaced values:
|
InterpolatedtoRaw | When you request interpolated data, you specify an interval or number of samples. If the actual stored number of raw samples is greater than required, you will get interpolated data as described above. If the actual number of stored samples are less than the required, then you will get the raw samples. In this way, the needs of trending detail and application load are balanced. This mode is best used when querying compressed data because the Data Archiver can switch to the more efficient raw data query. |
Lab | The Lab sampling mode only returns the collected values, without any interpolation of the value. The collected value is repeated for each interval until there is a change in the raw data sample's value. Lab sampling is most often used to create a step chart rather than a smooth curve. Use Lab sampling instead of interpolated if you only want true collected values returned. The Lab sampling mode is generally not useful on highly compressed data. Use interpolated sampling instead. |
LabtoRaw | LabtoRaw is an extension to Lab mode of sampling and similar to InterpolatedtoRaw mode where you will be switched to raw data or lab when the actual data samples are fewer than the requested samples. |
Trend | The Trend sampling mode was designed to produce maximum performance when retrieving data for plotting, particularly over long time periods. The trend sampling mode returns the maximum and minimum value collected during each interval. When plotted, this makes it possible to display an accurate representation of the data that wont miss any extrema, by only retrieving a minimum of points. For example, a trend of one year of data with a one-day interval will return 730 values consisting of the 365 minimums and 365 maximums for each day of the year. |
TrendtoRaw | The TrendtoRaw sampling mode almost always produces the same results as the Trend sampling mode. The exception is that, when more samples are requested than there are raw data points, the TrendtoRaw sampling mode returns all of the available raw data points with no further processing. TrendtoRaw is used when the number of actual data samples are fewer than the requested number of samples. In that case, TrendtoRaw retrieves raw data in a given interval (between a selected raw minimum and raw maximum). |
Trend2 | The Trend2 sampling mode is a modified version of the Trend sampling mode. Trend2 sampling splits up a given time period into a number of intervals (using either a specified number of samples or specified interval length), and returns the minimum and maximum data values that occur within the range of each interval, together with the timestamps of the raw values. The key differences between Trend and Trend2 sampling modes are in:
Trend sampling mode is more suitable for plotting applications that prefer evenly-spaced data. Trend2 sampling mode is more suitable for analysis of mins and maxes and for plotting programs that can handle unevenly spaced data. |
TrendtoRaw2 | The TrendtoRaw2 sampling mode is a modified version of the TrendtoRaw sampling mode. The TrendtoRaw2 sampling mode almost always produces the same results as the Trend2 sampling mode. The exception is that, when more samples are requested than there are raw data points, the TrendtoRaw2 sampling mode returns all of the available raw data points with no further processing. |
Calculated | Returns samples based on a selected Calculation mode. Refer to Calculation Modes for more information. |
RawByFilterToggle | RawByFilterToggle returns filtered time ranges. The values returned are 0 and 1. If the value is 1, then the condition is true and 0 means false. This sampling mode is used with the time range and filter tag conditions. The result starts with a starting time stamp and ends with an ending timestamp |
Calculation Modes
Calculation modes are used when the sampling mode is set to Calculated. The data type of all calculated values will be DoubleFloat except for MinimumTime, MaximumTime, FirstRawTime and LastRawTime which will be a Date. The datatype of the values of FirstRawValue and LastRawValue will be the same as that of the selected tag.
Calculation Mode | Results |
---|---|
Count | Displays the number of raw samples in the specified interval. This only indicates the count and does not display the actual values or qualities of the samples. The Count calculation mode is useful for analyzing the distribution of raw data samples. If you have a higher number of raw samples than expected, you may decide to implement collector or archive compression. If samples are missing, then you may want to slow your collection rates. |
State Count | Displays the number of times a tag has transitioned to another state from a previous state. A state transition is counted when the previous good sample is not equal to the state value and the next good sample is equal to state value. |
State Time | Displays the duration that a tag was in a given state within an interval. |
Minimum | Displays the minimum value in a specified interval with good data quality. This value may be raw or interpolated. Note: The Minimum and MinimumTime calculation retrieve two additional samples per interval; one is interpolated at the interval start time and the other is interpolated at the interval end time. These samples are used to determine the min or max just like any raw value.
|
MinimumTime | Displays the time stamp of the minimum value in a specified interval. See the note in Minimum for additional information. |
Maximum | Displays the maximum value in a specified interval. Note: The Maximum and MaximumTime calculation internally retrieve two additional samples per interval; one is interpolated at the interval start time and the other is interpolated at the interval end time. These samples are used in the min or max just like any raw or interpolated value.
|
MaximumTime | Displays the time stamp of the maximum value in a specified interval. See the note in Maximum for additional information. |
RawAverage | Displays the arithmetic average of the raw values in a specified interval with good data quality. This is useful only when a sufficient number of raw data values are collected. |
Average | Similar to RawAverage, but performs a special logic for time weighting and for computing the value at the start of the interval. This is useful for computing an average on compressed data. |
OPCQOr and OPCQAnd | The OPCQOr is a bit wise OR operation of all the 16 bit OPC qualities of the raw samples stored in the specified interval. The OPCQAnd is a bit wise AND operation of all the 16 bit OPC qualities of the raw samples stored in the specified interval. |
Total | Retrieves the time-weighted total of raw and interpolated values for each calculation interval. The collected value must be a rate per 24 hours. This calculation mode determines a count from the collected rate. |
RawTotal | Displays the arithmetic sum of raw values in a specified interval. |
StandardDeviation | Displays the time-weighted standard deviation of raw values for a specified interval. |
RawStandardDeviation | Displays the arithmetic standard deviation of raw values for a specified interval. |
TimeGood | Displays the amount of time (in milliseconds) during an interval when the data is of good quality and matches filter conditions if the filter tag is used. |
FirstRawValue | Returns the first good raw value for a specified time interval. |
FirstRawTime | Returns the timestamp of the first good raw for a specified time interval. |
LastRawValue | Returns the last good raw value for a specified time interval. |
LastRawTime | Returns the timestamp of the last good raw for a specified time interval. |
TagStats | Allows you to return multiple calculation modes for a tag in a single query. |
Query Modifiers
Query Modifiers are used for retrieving data that has been stored in the archive. They are used along with sampling and calculation modes to get a specific set of data.
Query Modifier | Results |
---|---|
ONLYGOOD | The ONLYGOOD modifier excludes bad and uncertain data quality values from retrieval and calculations. Use this modifier with any sampling or calculation mode but it is most useful with Raw and CurrentValue queries. All the calculation modes such as minimum or average exclude bad values by default, so this modifier is not required with those. |
INCLUDEREPLACED | Normally, when you query raw data from Proficy Historian, any values that have been replaced with a different value for the same timestamp are not returned. The INCLUDEREPLACED modifier helps you to indicate that you want replaced values to be returned, in addition to the currently retrievable data. However, you cannot query only the replaced data and the retrievable values that have replaced the other values. You can query all currently visible data and get the data that has been replaced. This modifier is only useful with rawbytime or rawbynumber retrieval. Do not use it with any other sampling or calculation mode. |
INCLUDEDELETED | The INCLUDEDELETED modifier retrieves the value that was previously deleted. Data that has been deleted from the archiver is never actually removed but is marked as hidden. Use the INCLUDEDELETED modifier to retrieve the values that were deleted, in addition to any non-deleted values during the query time period. This modifier is only useful with rawbytime or rawbynumber retrieval. Do not use it with any other sampling or calculation mode. |
ONLYIFCONNECTED ONLYIFUPTODATE |
The ONLYIFCONNECTED and ONLYIFUPTODATE modifiers can be used on any sampling or calculation mode to retrieve bad data if the collector is not currently connected and sending data to the archiver. The bad data is not stored in the IHA file but is only returned in the query. If the collector reconnects and flushes data and you run the query again, the actual stored data is returned in the following situations:
|
ONLYRAW | The ONLYRAW modifier retrieves only the raw stored samples. It does not add interpolated or lab sampled values at the beginning of each interval during calculated retrieval such as average or minimum or maximum. Normally, a data query for minimum value will interpolate a value at the start of each interval and use that together with any raw samples to determine the minimum value in the interval. Interpolation is necessary because some intervals may not have any raw samples stored. Use this query modifier with calculation modes only, not with raw or sampled retrieval like interpolated modes. |
LABSAMPLING | The LABSAMPLING modifier affects the calculation modes that interpolate a value at the start of each interval. Instead of using interpolation, lab sampling is used. When querying highly compressed data, you may have intervals with no raw samples stored. An average from 2 PM to 6 PM on a one-hour interval will interpolate a value at 2 PM, 3 PM, 4 PM, and 5 PM and use those in addition to any stored samples to compute averages. When you specify LABSAMPLING, then lab sampling mode is used instead of interpolated sampling mode to determine the 2 PM, 3 PM, 4 PM, and 5 PM values. A lab sampled average would be used when querying a tag that never ramps but changes in a step pattern such as a state value or setpoint. Use this query modifier with calculation modes only, not raw or sampled retrieval like interpolated modes. |
ENUMNATIVEVALUE | The ENUMNATIVEVALUE modifier retrieves the native, numeric values such as 1 or 2 instead of string values such as on/off for the data that has enumerated states associated with it. You can use ENUMNATIVEVALUE with any sampling or calculation mode. |
INCLUDEBAD | Normally, when you query calculated data from Historian, only good data quality raw samples are considered. INCLUDEBAD modifier includes bad data quality values in calculations. You can use INCLUDEBAD with any sampling or calculation mode. |
FILTERINCLUDEBAD | Typically, while filtering we use only good data quality values. When we use FILTERINCLUDEBAD, the bad data quality values are considered when filtering to determine time ranges. This query modifier is not always recommended. |
USEMASTERFIELDTIME | The USEMASTERFIELDTIME query modifier is used only for the MultiField tags. It returns the value of all the fields at the same timestamp of the master field time, in each interval returned. |
HONORENDTIME | Normally, a query keeps searching through archives until the desired number of samples has been located, or until it gets to the first or last archive. However, there are cases where you would want to specify a time limit as well. For example, you may want to output the returned data for a RawByNumber query in a trend screen, in which case there is no need to return data that would be offscreen. In cases where you want to specify a time limit, you can do this by specifying an end time in your RawByNumber query and including the HONORENDTIME query modifier. Since RawByNumber has direction (backward or forward), the end time must be older than the start time for a backward direction or newer than the start time for a forward direction. Use this query modifier only with the RawByNumber sampling mode. |
EXAMINEFEW | Queries using calculation modes normally loop through every raw sample, between the given start time and end time, to compute the calculated values. When using FirstRawValue, FirstRawTime, LastRawValue, and LastRawTime calculation modes, we can use only the raw sample near each interval boundary and achieve the same result. The EXAMINEFEW query modifier enables this. If you are using one of these calculation modes, you may experience better read performance using the EXAMINEFEW query modifier. Using this query modifier is recommended when:
Query performance varies depending on all of the above factors. Use this query modifier only with FirstRawValue, FirstRawTime, LastRawValue, and LastRawTime calculation modes. |
EXCLUDESTALE |
Stale tags are tags that have no new data samples within a specified period of time, and which have the potential to add to system overhead and slow down user queries. The EXCLUDESTALE query modifier allows for exclusion of stale tags in data queries. Unless permanently deleted, stale tags from the archiver are not removed but are simply marked as stale. Use the query without this query modifier to retrieve the sample values. Data is not returned for stale tags. An ihSTATUS_STALED_TAG error is returned instead. |
Filtered Data Queries
Filtered data queries enhance Historian by adding filter tags and additional filtering criteria to standard queries. Unfiltered data queries in Historian allow you to specify a start and end time for the query, then return all data samples within that interval. A filtered data query, however, will allow you to specify a condition to filter the results by, as well as calculation modes to perform on the returned data. Filtered data queries are performed on the Historian server.
For example, a filtered data query is useful when trying to retrieve all data for a specific Batch ID, Lot Number, or Product Code and for filtering data where certain limits were exceeded, such as all data where a temperature exceeded a certain value. Rather than filtering a full day's worth of process data in the client application, you can filter data in the Historian archiver, and only return the matching results to the client application. The result is a smaller, more relevant data set.
You can use filter criteria with raw, interpolated, and calculated sampling modes. You cannot use it with current value sampling. The logic of selecting intervals is always interpolated, even when the data retrieval is raw or calculated. The value that triggers a transition from false to true can be a raw value or interpolated value.
You cannot use a filtered data query in an iFIX chart. For more information, refer to Advanced Topics section in the online help.
Filter Parameters for Data Queries
Use of filter parameters with a data query is optional.
Parameter | Description |
---|---|
Filter Tag | The single tag name used when applying the filter criteria. You can enter your filter conditions using Filter tag, Filter Comparison Mode, and Filter Comparison Value or you can put that all that information in a single FilterExpression. |
Filter Expression | An expression which includes one or more filter conditions. The type of conditions used are:
Filter Expression can be used instead of FilterTag, FilterComparisonMode and FilterValue parameters. While using FilterExpression, the expression is passed within single quotes and for complex expressions we write the conditions within a parenthesis. There is no maximum length for a filter expression, but if it is called using OLEDB or Excel, they may have their own limitations. |
Filter Mode | The type of time filter. The Filter Mode defines how time periods before and after transitions in the filter condition should be handled. For example, AfterTime indicates that the filter condition should be True starting at the timestamp of the archive value that triggered the True condition and leading up to the timestamp of the archive value that triggered the False condition.
|
Filter Comparison Mode | Filter Comparison Mode is only used if Filter Tag is filled in. The Filter Comparison Mode defines how archive values for the Filter Tag should be compared to the Filter Value to establish the state of the filter condition. If a Filter Tag and Filter Comparison Value are supplied, time periods are filtered from the results where the filter condition is False. The type of comparison to be made on the filter comparison value:
|
Filter Comparison Value | Filter Comparison Value is only used if Filter Tag is filled in. The value to compare the filter tag with when applying the appropriate filter to the data record set query (to determine the appropriate filter times). |
Filtered Queries in the Excel Add-in Example
batchid
and ramp. The batchid
tag is updated before a new batch is produced with the new batch's ID. The ramp
tag contains raw data sent by a device in the process. In this example, it is requested that Historian return data samples at ten second intervals for the ramp
tag during the period that the batchid
tag is set to B1.Time Stamp | Value | Data Quality |
---|---|---|
07/30/2003 08:00:10 | 16 | Good |
07/30/2003 08:00:20 | 22 | Good |
07/30/2003 08:00:30 | 34 | Good |
07/30/2003 08:00:40 | 46 | Good |
07/30/2003 08:00:50 | 50 | Good |
07/30/2003 08:01:00 | 55 | Good |
batchid
tag for the same time interval, we would receive the following results:
Time Stamp | Value | Data Quality |
---|---|---|
07/30/2003 08:00:00 | B0 | Good |
07/30/2003 08:00:20 | B1 | Good |
07/30/2003 08:00:45 | B2 | Good |
Filtering Data Queries in the Excel Add-in
To find the values of the ramp
tag for the B1 batch, enter the following values into the Historian Filtered Data Query dialog box:
- In the
Tag Name(s)
field, enter the tag you want to receive results from - theramp
tag in this example. - Select a start and end time for your query.
- In the
Filter Tag
field, enter the tag you want to enable filtering with -batchid
in this example. - In the
Filter Comparison
field, select your comparison condition. - n the
Include Data Where Value Is
field, enter your filter condition value. - In the
Include Times
field, select your filter mode. - In the
Sampling Type
field, select your sampling mode. - In the
Calculation
field, select your calculation mode. - Select your
Sampling Interval
. - In the
Output Display
field, select the tag values you want to display.