### Create Continuous Parser using Whistle Script Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/deployment/basic-configuration_hl=zh-cn This example details the configuration of a 'basic-continuous' parser through the MDE web interface using Whistle script. The script is tailored for continuous data, capturing start and end timestamps, duration, and relevant metadata. ```whistle package mde [ { tagName: $root.tagName; timestamps: MapTimestamp($root); data: MapData($root); duration: $root.duration; embeddedMetadata: $root[where ($.field != "value" and $.field != "tagName" and $.field != "startTimestamp" and $.field != "endTimestamp" and $.field != "messageId")]; }] def MapTimestamp(input) { continuousTimestamp: { eventTimestampStart: input.startTimestamp; eventTimestampEnd: input.endTimestamp; }; } def MapData(input) { complex: input.value; } ``` -------------------------------- ### Server-Side Downsampling Configuration Example Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/guides/access/how-to-query-in-fed-api_hl=ja Shows how to configure server-side downsampling for data retrieval from Bigtable and BigQuery. This example specifies a 10-minute interval with a MEAN aggregation. ```url-query-string sample=[10,MINUTE,MEAN] ``` -------------------------------- ### Sample GET Request for BigQuery Records Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/guides/access/how-to-query-in-fed-api_hl=ja Demonstrates a sample GET request to the Federation API for retrieving records from BigQuery. It includes query parameters to enable metrics, headers, and metadata in the response. ```http GET http://localhost:8080/data/v1/bigquery/records/primepaintingrobot-01-airhumidity/default-numeric-records/latest?includeMetrics=true&includeHeader=true&includeMetadata=true ``` -------------------------------- ### Release Signature Example (v1.3.4) Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/release-notes_hl=fr This example presents a release signature for version 1.3.4. It follows a standard format including a numerical identifier, the version string, and a hexadecimal checksum for integrity verification. The provided signature includes '3374a74', '1.3.4', and 'e7ea042c2381ab7275d681e66a8c0f8a'. ```text 3374a74 1.3.4 e7ea042c2381ab7275d681e66a8c0f8a ``` -------------------------------- ### Create Continuous Parser using REST API Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/deployment/basic-configuration_hl=zh-cn This snippet illustrates creating a 'basic-continuous' parser via the REST API. It specifies the parser's configuration and includes a Whistle script designed to handle continuous data, mapping fields like tag name, timestamps (start and end), duration, and metadata. ```json { "name": "basic-continuous-to-basic-continuous", "messageClassName": "basic-continuous", "typeReference": { "name": "basic-continuous", "version": 1 }, "script": "package mde\n\n[{\ntagName: $root.tagName;\ntimestamps: MapTimestamp($root);\ndata: MapData($root);\nduration: $root.duration;\nembeddedMetadata: $root[where ($.field != \"value\" and $.field != \"tagName\" and $.field != \"startTimestamp\" and $.field != \"endTimestamp\" and $.field != \"messageId\")];\n}]\n\ndef MapTimestamp(input) {\n continuousTimestamp: {\n eventTimestampStart: input.startTimestamp;\n eventTimestampEnd: input.endTimestamp;\n };\n}\n\ndef MapData(input) {\n complex: input.value;\n}\n" } ``` -------------------------------- ### Create Basic Continuous Parser using Whistle Script Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/deployment/basic-configuration_hl=fr Sets up a parser for basic continuous data. This configuration includes the parser name, message class, and a Whistle script for mapping data fields such as tagName, timestamps, duration, and embedded metadata. The script defines how to extract start and end event timestamps and complex data values. ```json { "name": "basic-continuous-to-basic-continuous", "messageClassName": "basic-continuous", "typeReference": { "name": "basic-continuous", "version": 1 }, "script": "package mde\n\n[{\"tagName\": $root.tagName;\ntimestamps: MapTimestamp($root);\ndata: MapData($root);\nduration: $root.duration;\nembeddedMetadata: $root[where ($.field != \"value\" and $.field != \"tagName\" and $.field != \"startTimestamp\" and $.field != \"endTimestamp\" and $.field != \"messageId\")];\n}]\n\ndef MapTimestamp(input) {\n continuousTimestamp: {\n eventTimestampStart: input.startTimestamp;\n eventTimestampEnd: input.endTimestamp;\n };\n}\n\ndef MapData(input) {\n complex: input.value;\n}\n" } ``` ```whistle package mde [{ tagName: $root.tagName; timestamps: MapTimestamp($root); data: MapData($root); duration: $root.duration; embeddedMetadata: $root[where ($.field != "value" and $.field != "tagName" and $.field != "startTimestamp" and $.field != "endTimestamp" and $.field != "messageId")]; }] def MapTimestamp(input) { continuousTimestamp: { eventTimestampStart: input.startTimestamp; eventTimestampEnd: input.endTimestamp; }; } def MapData(input) { complex: input.value; } ``` -------------------------------- ### Install Manufacturing Connect Load Balancer Helm Chart Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/deployment/external-lb-for-mc Installs the `mc-http-lb` Helm chart in the current directory. This command provisions the External HTTP Load Balancer with a Google-managed SSL certificate. ```bash helm install mc-http-lb . ``` -------------------------------- ### JSON Schema Example for Metadata Bucket Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/concepts/metadata_hl=de This example demonstrates a JSON schema for a metadata bucket, specifying required properties and their types. MDE automatically extends this schema by adding 'additionalProperties: false' to enforce strict instance structure. ```json { "$schema": "https://json-schema.org/draft/2019-09/schema#", "type": "object", "properties": { "deviceName": { "type": "string" } }, "required": ["deviceName"] } ``` -------------------------------- ### Release Signature Example Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/release-notes_hl=fr This code block displays a typical release signature format. It includes a version number and a hexadecimal hash, commonly used for verifying the integrity and authenticity of a software release. The example shows '8584357' for the version and 'db04eff8fb5f0d2267d1653498480ec7' as the signature hash. ```text 8584357 1.3.5 db04eff8fb5f0d2267d1653498480ec7 ``` -------------------------------- ### Initialize Looker Project with Template Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/guides/analyze/how-to-set-up-a-new-mde-looker-project Navigates to the newly created repository, checks out the master branch, adds the MDE Looker Project Template as a remote, pulls the template files, and pushes them to the origin master branch. ```bash cd mde-looker git checkout -b master git remote add project https://github.com/GoogleCloudPlatform/mde-looker-project-template.git git pull project master git push -u origin master ``` -------------------------------- ### Get Records from BigQuery Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/guides/access/how-to-query-in-fed-api Retrieves the latest records for a given set of tags and a type combination from BigQuery. Supports filtering by start and end timestamps and pagination. ```APIDOC ## GET /data/v1/bigquery/records/{typeName} ### Description Retrieves records from BigQuery for a specified type, optionally filtered by tags and a time range. ### Method GET ### Endpoint `/data/v1/bigquery/records/{typeName}` ### Parameters #### Query Parameters - **tags** (string, comma-separated) - Required - Comma-separated list of tags to filter records by. - **startTimestamp** (long) - Required - The start of the time range in milliseconds since the epoch. - **endTimestamp** (long) - Required - The end of the time range in milliseconds since the epoch. - **pageSize** (integer) - Optional - The maximum number of records to return per page. ### Request Example ``` GET http://localhost:8080/data/v1/bigquery/records/default-numeric-records?tags=station-a-powerusage,station-b-powerusage&startTimestamp=1691057738790&endTimestamp=1691061338790&pageSize=100 ``` ### Response #### Success Response (200) - **header** (object) - Contains metadata about the response, such as timestamps and counts. - **tags** (array) - An array of tag objects, each containing its data. - **metrics** (object) - Performance metrics for the request. #### Response Example ```json { "header": { "startTime": "1691061786759", "endTime": "1691061786759", "tagCount": 1, "dataRecordCount": 1, "dataFields": { "d": "duration", "dp": "discretePayload", "cp": "continuousPayload", "its": "ingestTimestamp", "cmr": "cloudMetadataRef", "sdi": "sourceMessageId", "np": "numericPayload", "id": "id", "v": "value", "em": "embeddedMetadata", "et": "eventTimestampEnd", "mcm": "materializedCloudMetadata", "t": "eventTimestamp", "st": "eventTimestampStart" } }, "tags": [ { "tagName": "primepaintingrobot-01-airhumidity", "data": [ { "t": "1691061786759000", "np": { "v": 81.79 }, "em": "{\"datatype\":\"float\",\"description\":\"This is a fake prime painting robot on NodeRed\",\"deviceID\":\"75c18751-7a94-453e-86f5-67be2b0c8fd4\",\"deviceName\":\"primepaintingrobot-01\",\"headers\":{},\"messageClassName\":\"default-numeric-value\",\"metadata\":{\"brand\":\"Philips\",\"measurementUnit\":\"percentage\",\"sensorType\":\"humidity\",\"shift\":\"Shift 4\"},\"registerId\":\"a2635b62-0e89-40ff-b172-13eea37a92c2\",\"success\":true,\"unit\":\"percentage\"}", "mcm": "{}", "cmr": "{}" } ], "typeName": "default-numeric-records" } ], "metrics": { "totalTimeMS": 2371 } } ``` ``` -------------------------------- ### Get records from BigQuery by tag and time range Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/guides/access/how-to-query-in-fed-api_hl=zh-cn Retrieves numeric records from BigQuery based on tags and a specified time range. Requires tag names, start and end timestamps, and an option to include headers. ```HTTP GET http://localhost:8080/data/v1/bigquery/tags/default-numeric-records?startTimestamp=1690942297865&endTimestamp=1691062557865&includeHeader=true ``` -------------------------------- ### Terraform Apply Success Message Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/deployment/upgrade-guide_hl=de Example of a successful Terraform apply completion message. It indicates the number of resources added, changed, or destroyed during the deployment process. ```bash Apply complete! Resources: 1 added, 34 changed, 0 destroyed. ``` -------------------------------- ### Create Basic Discrete Parser using MDE Web Interface Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/deployment/basic-configuration_hl=de This example demonstrates the steps to create a basic discrete parser through the MDE web interface. It involves filling in parser details and pasting a Whistle script to process discrete data. ```whistle package mde [{ tagName: $root.tagName; timestamps: MapTimestamp($root); data: MapData($root); embeddedMetadata: $root[where ($.field != "value" and $.field != "tagName" and $.field != "timestamp" and $.field != "messageId")]; }] def MapTimestamp(input) { eventTimestamp: input.timestamp; } def MapData(input) { complex: input.value; } ``` -------------------------------- ### POST /configuration/v1/message-classes (continuous) Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/deployment/basic-configuration_hl=ko This endpoint allows you to create a new message class for continuous data. It defines the criteria for identifying continuous events, often characterized by start and end timestamps. ```APIDOC ## POST /configuration/v1/message-classes (continuous) ### Description Creates a message class for identifying continuous data events. This class is suitable for data that spans a time interval, indicated by start and end timestamps. ### Method POST ### Endpoint /configuration/v1/message-classes ### Parameters #### Request Body - **name** (string) - Required - The name of the message class, e.g., `basic-continuous`. - **priority** (integer) - Required - The priority of the message class for processing order. - **expression** (string) - Required - A DRL expression to evaluate incoming events for this message class. Example: `#root.event['value'] != null && !(#root.event['value'] instanceof T(Number)) && #root.event['startTimestamp'] != null && #root.event['endTimestamp'] != null && #root.event['tagName'] != null` ### Request Example ```json { "name": "basic-continuous", "priority": 1950, "expression": "#root.event['value'] != null && !(#root.event['value'] instanceof T(Number)) && #root.event['startTimestamp'] != null && #root.event['endTimestamp'] != null && #root.event['tagName'] != null" } ``` ### Response #### Success Response (201 Created) - **message** (string) - Confirmation message indicating the message class was created successfully. #### Response Example ```json { "message": "Message class 'basic-continuous' created successfully." } ``` ``` -------------------------------- ### Create Continuous Message Class in MDE (Web Interface) Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/deployment/basic-configuration_hl=ja This snippet guides through creating a 'basic-continuous' message class using the MDE web interface. It involves adding a new message class, setting its name to 'basic-continuous', priority to 1950, and entering the provided expression. This expression is designed to identify continuous data events with start and end timestamps, and a tag name. ```plaintext 1. On the MDE web interface, click **Message Classes**. 2. Click **Add new Message Class**. 3. Fill in the **Message Class name** as `basic-continuous`. 4. Copy the following expression and paste it on the **Expression** box: ``` #root.event['value'] != null && !(#root.event['value'] instanceof T(Number)) && #root.event['startTimestamp'] != null && #root.event['endTimestamp'] != null && #root.event['tagName'] != null ``` 5. Change the priority to `1950`. 6. Click **Create**. ``` -------------------------------- ### Get Latest Records from BigQuery Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/guides/access/how-to-query-in-fed-api_hl=fr Retrieves the latest records for specified tags and a type combination from BigQuery. It requires the API endpoint, tag names, start and end timestamps, and a page size. The response contains header information, tag data with timestamps and payloads, and metrics. ```HTTP GET http://localhost:8080/data/v1/bigquery/records/default-numeric-records?tags=station-a-powerusage,station-b-powerusage&startTimestamp=1691057738790&endTimestamp=1691061338790&pageSize=100 ``` -------------------------------- ### Create Basic Continuous Parser using MDE Web Interface Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/deployment/basic-configuration_hl=de This example provides instructions for creating a basic continuous parser using the MDE web interface. It details the fields to fill and the Whistle script required for processing continuous data, including duration and timestamps. ```whistle package mde [{ tagName: $root.tagName; timestamps: MapTimestamp($root); data: MapData($root); duration: $root.duration; embeddedMetadata: $root[where ($.field != "value" and $.field != "tagName" and $.field != "startTimestamp" and $.field != "endTimestamp" and $.field != "messageId")]; }] def MapTimestamp(input) { continuousTimestamp: { eventTimestampStart: input.startTimestamp; eventTimestampEnd: input.endTimestamp; }; } def MapData(input) { complex: input.value; } ``` -------------------------------- ### Create Service Account and Key for Looker Source: https://docs.cloud.google.com/manufacturing-data-engine/docs/guides/analyze/how-to-set-up-a-new-mde-looker-project Creates a service account for Looker to access BigQuery data, grants necessary IAM roles, and generates a JSON key file. The key file location and service account details are outputted for Looker configuration. ```bash export SA_LOOKER="mde-looker" export PROJECT_ID=$(gcloud config get-value project) gcloud iam service-accounts create $SA_LOOKER --display-name "MDE Looker account" gcloud projects add-iam-policy-binding ${PROJECT_ID} --member serviceAccount:${SA_LOOKER}@${PROJECT_ID}.iam.gserviceaccount.com --role "roles/bigquery.dataEditor" gcloud projects add-iam-policy-binding ${PROJECT_ID} --member serviceAccount:${SA_LOOKER}@${PROJECT_ID}.iam.gserviceaccount.com --role "roles/bigquery.jobUser" export SA_LOOKER_KEY=~//mde-projects/$PROJECT_ID/${SA_LOOKER}_key.json gcloud iam service-accounts keys create $SA_LOOKER_KEY --iam-account ${SA_LOOKER}@${PROJECT_ID}.iam.gserviceaccount.com cat <