Time Series Data Ingestion

About Time Series Data Ingestion

The Predix Essentials Time Series Data module provides POST methods, JSON payload and CSV file, to ingest time series data. The CSV supported formats are ODBExportCSV, and ProficyFileCollectorCSV. This module is used by various features and other modules to consume time series data for display, charting, and analysis.

The two ingestion methods can be used for one or more tags. For each tag, a series of data points are defined by timestamp, data value, and data quality indicator. The data points are specified as numeric, string or boolean values.

After creating the ingestion files, they are submitted through REST endpoints and can be queried using the time series data retrieval endpoint to ensure correct data ingestion. Correct any errors in the ingestion file by resubmitting as needed, which replaces the existing data. Any data with new value type or quality indicator ingests as new records.

Note: All CRUD operations are performed within the context of a tenant. Every request header must include the authorization header and the tenant UUID.

The following definitions are helpful for understanding the headers and parameters:

Authorization
Enter the token_type followed by a space, then the access_token that is in the response you previously obtained, for example, bearer eyJhbGciOiJSUzI1NiJ9.A... The following code sample includes a bearer token:
{
  "access_token": "eyJhbGciOiJSUzI1NiJ9.A...",
  "token_type": "bearer",
  "refresh_token": "eyJhbGciOiJSUzI1NiJ9.e..",
  "expires_in": 86399,
  "scope": "openid",
  "jti": "5e2c90ea-9edd-4da1-80c8-4a0d67f77ef3"
}
Content-type
The MIME type of the body of the request. Supported types for ingestion are application/x-www-form-urlencoded, and application/json
tenant
The unique ID for the tenant (for example, 07F28C049E0F4F29B8E85E4A6C916D7F)
file_type
The file type you are using to ingest time series data. It is a required parameter. The permitted values are json, X-GE-CsvFormat, and ProficyFileCollectorCSV
dataSource
The data source you are using for time series data. This parameter is optional. If provided, overrides the default data source defined in the configuration file. For example, predixtss.PredixTSS is a time series data source.
200
On successful ingestion, a 200 OK message appears.

Ingest Time Series Data

Before You Begin

Note: The maximum number of concurrent ingestion processes is 10. Additional processes are enqueued until threads become available.
You need the following files and information:
  • Time series data stored JSON, ZIP or CSV file format.
    Note: The maximum file size per ingestion is 50MB.
    Make sure that the time stamp provided in the CSV file is in the YYYY-MM-DDTHH:MM:SS.000 format.
  • Supported numeric, string, and Boolean tag values.
  • OAuth token used to ingest time series data.
    Important: Make sure you have a valid, unexpired token. Tokens are client-specific and expire within a specified time. Check the JSON response for the token expiration.
  • In the Setup section, obtain the following information:
    • Client ID
    • Time Series Data ingestion URL
    • User name
  • Access to a REST client such as Postman or Advanced REST client.
Note: Time Series Data performs a check to ensure users can only query tags that are associated with assets they have permission to access. If tags are not associated with a Predix Essentials asset, or permission is not configured properly, users can no longer query time series data using Time Series Data.
The following table describes the value and their definitions as it relates to quality:
ValueDescription
0Bad quality.
1Uncertain quality.
2Not applicable.
3Good quality. If you do not specify quality, the default is good (3).

Procedure

  1. In the REST client, enter the ingestion URL in the HOST value field. Append the required query parameters. For example, to ingest ODB format into Time Series Data, the query parameters are dataSource=predixtss.PredixTSS&file_type=ODBExportCSV.
    Note: You can obtain this URL in the Setup section.
  2. Select POST.
  3. Add the required headers.
  4. In the Body field, perform one of the following actions depending on the content-type of the raw request.
    If file_type isDo this
    ODBExportCSV
    1. Select form-data.
    2. Select CSV file from the drop-down list box.
    3. Paste the time series data in text format in the Body field.

      The following code sample shows a time series in comma-separated format as exported from an Oracle database:

      Time,298936.AUTOMATION,298936.AFPAP,298936.AFPCS,298936.AFPEP,298936.CA_CRT 9/15/15 
2:22:02 PM,182.4691925,30.22567177,3.45894742,0,96.1139679 9/15/15 
2:27:02 PM,182.1760254,30.22491646,3.397117376,0,96.15263367 9/15/15 
2:32:02 PM,182.2230225,30.22415924,3.405554295,0,96.19130707 9/15/15 
2:37:02 PM,181.9922791,30.22340202,3.410028934,0,96.22998047
    JSON
    1. Select raw
    2. Select JSON (application/json) from the drop-down list box.
    3. Paste the time series data in JSON format in the Body field.

      The following code sample shows time series data with multiple tags:

      { "tags": [ { "tagId": "TagLength", "data": [ { "ts": "2015-04-13T18:10:00.000-0000", "v": "1.123", "q":
      "3" }, { "ts": "2015-04-13T18:11:00.000-0000", "v": "3.223", "q": "3" }, {
      "ts": "2015-04-13T18:12:00.000-0000", "v": "1.323", "q": "3" }, { "ts":
      "2015-04-13T18:13:00.000-0000", "v": "3.423", "q": "3" }, { "ts":
      "2015-04-13T18:14:00.000-0000", "v": "1.523", "q": "3" }, { "ts":
      "2015-04-13T18:15:00.000-0000", "v": "", "q": "3" } ] }, { "tagId":
      "TagPressure", "data": [ { "ts": "2015-04-13T18:00:00.000-0000", "v": 2.231, "q":
      "3" }, { "ts": "2015-04-13T18:01:00.000-0000", "v": 2.232, "q": "3" }, { "ts":
      "2015-04-13T18:02:00.000-0000", "v": 2.233, "q": "3" }, { "ts":
      "2015-04-13T18:03:00.000-0000", "v": 2.234, "q": "3" }, { "ts":
      "2015-04-13T18:04:00.000-0000", "v": null, "q": "3" }, { "ts":
      "2015-04-13T18:05:00.000-0000", "v": null, "q": "3" } ] }, { "tagId":
      "Tag_Temperatue", "data": [ { "ts": "2015-04-13T18:10:00.000-0000", "v": "1.123",
      "q": "3" }, { "ts": "2015-04-13T18:11:00.000-0000", "v": "3.223", "q": "3" },
      { "ts": "2015-04-13T18:12:00.000-0000", "v": "1.323", "q": "3" }, { "ts":
      "2015-04-13T18:13:00.000-0000", "v": "3.423", "q": "3" }, { "ts":
      "2015-04-13T18:14:00.000-0000", "v": "1.523", "q": "3" }, { "ts":
      "2015-04-13T18:15:00.000-0000", "v": "3.623", "q": "3" } ] } ] }
    ProficyFileCollectorCSV
    1. Select form-data
    2. Select File from the drop-down list box.
    3. Select Choose Files to attach a valid CSV file for ingestion.

    The following code sample shows valid time series data in plain-text format (comma-separated values from a Proficy Historian file collector export):

    1445752668,BFP_PenDemo.Tag_Length,13.20765,Double,GOOD
1445752768,BFP_PenDemo.Tag_Length,18.782133,Double,GOOD
1445752868,BFP_PenDemo.Tag_Length,36.648777,Double,GOOD
1445752968,BFP_PenDemo.Tag_Length,36.648777,Double,GOOD
1445753068,BFP_PenDemo.Tag_Length,36.648777,Double,GOOD
1445753168,BFP_PenDemo.Tag_Length,36.648888,Double,GOOD
    In this sample, each row shows the following data sequence:
    1. Timestamp in UNIX EPOCH format (for example, 1445752668).
    2. Tag name (for example, BFPPenDemo.TagLength).
    3. Tag value (for example: 13.20765).
    4. Data type (for example, Double).
    5. Data quality (for example, good).
  5. Select Send.

Results

Upon successful request, a 201 created message appears.

Verify Time Series Ingestion

After ingesting time series data using the REST endpoint, you can use HTTP POST, cURL, or Predix Essentials to verify proper ingestion.

Before You Begin

You must have ingested the asset model with tag data and ingested time series data for the previously ingested tags in the asset model using the POST method for a specific tenant. For that tenant, you must have access permission for assets you wish to query and access time series data.

About This Task

You can use one of the following ways to retrieve the ingested time series data:
  • Use HTTP POST method.
  • Use the cURL command to retrieve data.
  • Plot tags using the Analysis tab in the Predix Essentials application.

Procedure

  • Using the HTTP POST method
    1. In the REST client, select POST.
    2. Enter the retrieval URL in the HOST value field.
    3. Add the required headers.
    4. In the Body field, select raw.
    5. Select JSON (application/json) from the drop-down box.
    6. Paste the JSON format in the Body field with the required retrieval parameters. Make sure you provide the tags list, start and end time to retrieve specific the ingested time series data.
    On successful retrieval, you receive 200 OK and tag data in requested format.
  • Using the cURL command
    1. Access the command prompt.
    2. Run a cURL command as shown in the example below to retrieve the required time series data.
      curl 'https://api.example.com/v3/time_series/query' -i -X POST -H 'Content-Type: application/json' -H 'accept: application/json' -H 'accept: application/json' -H 'Authorization: <Authorization>' -H 'tenant: <tenant>' -d '{
  "tagList" : [ {
    "tagId" : "Inverter1-ASSET-TYPE1.Tag_Pressure",
    "unitsTo" : "psi",
    "attributes" : {
      "kay1" : [ "value1" ],
      "key2" : [ "value2" ]
    },
    "groups" : [ {
      "name" : "attribute",
      "attributes" : [ "Key1", "key2" ]
    } ]
  }, {
    "tagId" : "Inverter1-ASSET-TYPE2.Tag_Temperature",
    "unitsTo" : "c"
  }, {
    "tagId" : "Inverter1-ASSET-TYPE3.Tag_Length"
  } ],
  "operation" : "raw",
  "startTime" : "2016-02-09T15:00:00.000Z",
  "endTime" : "2016-02-09T16:00:00.000Z",
  "sampleCount" : 300,
  "useOnlyGoodData" : true,
  "direction" : "forward",
  "responseFormat" : "PROFICYLIKE",
  "precisionLevel" : "3",
  "useOnlyNumericData" : true
}
    On successful retrieval, you receive 200 OK and tag data in requested format.
  • Use Predix Essentials to plot tags
    1. Log into Predix Essentials.
    2. In the module navigation menu, select Essentials > Analysis.
    3. Navigate to the specific asset from the Context Browser.
    4. Select New Ad Hoc Analysis from the View Selector. The Ad Hoc Analysis view displays with no tags plotted.
    5. Use the Date Picker to select a date and time range.
      Note: Select a date and time-range around when the time series data was ingested into the tenant to obtain the resulting time-series data for the specific asset tags.
    6. To plot tags from the search list to the chart:
      1. Select , and enter at least three characters.
        Tip: The search allows you to find a tag that includes the characters sequentially entered.
      2. Select a tag in the search list, drag it over a chart, and then drop it on the chart. The selected chart is highlighted and framed as you drop the tag. If the tag is already plotted on the selected chart, Predix Essentials displays an error message.
      3. Close the Search field to access the list of plotted tags.
    7. Select List, and then select Save as Template.
    8. Type a unique label for the template, and then select Save.
    Your analysis view is saved to the selected asset. You can use the View drop-down list box to select the saved view. This way you do not have to plot tags again. Predix Essentials allows tags with no data for a specified time range to be plotted to a chart. The tag displays in the plotted tags list, but is disabled. indicates that no data is available. Make sure to change the date range to when the time series data was ingested to include tag data. This way the tag displays as enabled and plots its data on the analysis chart.

Predix Time Series Data Bulk Ingestion

You need to work with the Time Series Data team to ingest Predix time series data in bulk. This module is in limited release.

Stream Time Series Data

To ingest time series data into Predix Essentials using a web socket, a ServiceOps ticket is needed to get the credentials to enable time series data streaming. To perform this task, do the following:

Procedure

  1. File http://sc.ge.com/*apmtimeseriesstreaming to request information to enable data streaming for your tenant. The ticket provides zone ID, web socket URL, scope, client ID, and secret.
  2. Follow the https://www.ge.com/digital/product-documentation to initiate web socket streaming.