googlecloudplatform/node-red-contrib-google-cloud

Node-RED nodes for Google Cloud Platform

node-red-contrib-google-cloud (BETA)

This is not an official Google product.

Node-RED nodes for Google Cloud Platform.

Node-RED is an open source project which provides a visual programming environment and runtime for execution of event based applications. Written in Node.js JavaScript, it executes on a wide variety of platforms including workstations, IoT devices and, most importantly for our discussion, on Google Cloud Platform environments which includes Compute Engines and Kubernetes.

The Node-RED story is based on discrete nodes of function which are dragged and dropped from a palette to a canvas and then wired together. This resulting flow of execution is then triggered by the arrival of external events including REST requests, Pub/Sub messages, timers and more.

One of the key reasons that Node-RED has become as popular as it has is the ease with which developers can build additional nodes that encapsulate rich sets functions. Once written, these add-on nodes can be used by flow writers without having to know the complexities of their underlying operation. One just drags a new node onto the canvas and uses it.

This project provides a set of core Google Cloud Platform oriented Node-RED nodes. Once installed, the capability to use a broad set of GCP services are immediately unlocked.

Installation

There are multiple ways to install node-red-contrib-google-cloud. The official ways are described in the Node-RED documentation. The name of the package for installation is node-red-contrib-google-cloud.

Google Cloud Credentials

Each of the new nodes made available through this package will communicate with the Google Cloud Platform (GCP). These interactions must be performed securely and require authentication information to be passed.

If Node-RED is running under a GCP environment such as a Compute Engine, Google Kubernetes Engine or Cloud Run then there is an implicit identity presented to GCP and the flow developer need do nothing special for authentication configuration. However, if your Node-RED runtime is not running under GCP or you wish to call a service with adistinct identity, then you will need to use explicit credentials.

On each node, we have the opportunity to supply credentials. These can be supplied either as a path to a named key file or by creating a Node-RED managed name credentials secret. Each credential defined as a Node-RED secret has the following properties:

Property	Type	Description
name	string	Label for easy identification, essentially a comment.
account	string	Credentials in the form of a JSON key.

The credentials for a service account can be acquired from the APIs & Services menu. After you finish creating a service account key, it will be downloaded in JSON format and saved to a local file. Copy and paste the contents of the file directly into the Key field in the node editor.

An alternative to supplying credentials through the Node-RED credentials mechanism is to point individual GCP nodes to a JSON key file that exists on the file system on which Node-RED is running. This capability was introduced to support the Kubernetes "secrets" architecture. A Kubernetes administrator could obtain a JSON key file and save that as a secret in the Kubernetes environment. Containers within the environment could then see the secrets as mounted files. This level of indirection and abstraction allows us to build images which don't contain the keys hard-coded into the images but yet make those keys available at run-time. In addition, if we need to change or rotate the keys, we can do so without having to rebuild any images.

Installation outside of GCP

When we run Node-RED on GCP compute resources such as Compute Engine or GKE, the environment to make GCP API service calls is already present. If we run Node-RED outside of GCP (for example on a desktop PC, an on-premises server or a Raspberry Pi) then some additional setup to connect and use GCP APIs is required. Specifically, an environment variable called GOOGLE_CLOUD_PROJECT must be set and be present in the environment in which Node-RED runs. The value of this variable should be the GCP project that you are going to interact with.

If you are running Node-RED from the command line, you can use:

export GOOGLE_CLOUD_PROJECT=<YourProjectId>
node-red

You can also permanently set the variable by editing /etc/environment and adding a line which reads:

GOOGLE_CLOUD_PROJECT=<YourProjectId>

Note: For Raspberry Pi users - There is the option of starting Node-RED automatically through the Linux systemctl daemon. This system does not use global environment variables and thus you must explicitly define the variable to use.

Edit the file /lib/systemd/system/nodered.service

and add a line which reads:

Environment="GOOGLE_CLOUD_PROJECT=<YourProjectId>"

within the [Service] section of the file.

The Google Cloud Platform Node-RED nodes

The set of Node-RED nodes are found in the GCP section of the palette. The current set of nodes are:

• PubSub In - Receive a published message.
• PubSub Out - Publish a new message.
• Google Cloud Store Read - Read from the object store.
• Google Cloud Store Write - Write to the object store.
• Log - Write an entry to Stackdriver logging.
• Tasks - Create a new Cloud Task instance.
• Entities - Perform Natural Language analysis of entities on a piece of text.
• Sentiment - Perform Natural Language analysis of sentiment on a piece of text.
• Vision - Analyse an image.
• Metadata - Retrieve the compute engine metadata.
• Monitoring - Write a new monitoring record.
• Speech to Text - Recognize speech from audio.
• Text to Speech - Encode text as spoken audio.
• Data loss prevention - Mask out identifiable information.
• Translation - Translate text from one language to another.
• BigQuery - Execute a BigQuery query.
• BigQuery Insert - Execute a BigQuery insert.
• IoT Command Send - Send an IoT command to a device.
• IoT Config Update - Update the config of an IoT device.
• IoT Telemetry Send - Send a telemetry message from a device.
• AutoML - Execute an AutoML prediction.
• DNS Change Create
• DNS Records List
• GCS Signed URL
• Document AI - Parse the contents of a PDF or image of a document.
• Spanner - Execute a SQL statement against Spanner.

We are very open to receiving feedback on additional GCP nodes that may be of value. Don't hesitate to open an issue should you have a desire for incorporating additional GCP functions.

See also:

• Blog: Low code programming with Node-RED comes to GCP - 2020-02
• Medium: GCP Document AI and Node-RED - 2021-02
• Medium: Node-RED solution deployment on GCP - 2021-03

node-red-contrib-google-cloud's Issues

languageCode on speech-to-text.js

Could the language parameter be left as setting or out of code for Google Recognition to try to identify the language by itself?
As it is in the repository code with "languageCode": "en-US", recogntion only works for en-US!

Here's the highlighted code part (speech-to-text.js line 53):
const config = {
"encoding": encoding,
"sampleRateHertz": sampleRateHertz,
"languageCode": "en-US"
}

Specify a file name in the configuration for GCS Read

Currently, we specify the file name that we wish to read through the GCS read node using msg.filename. This issue asks for an enhancement to be able to specify a file name in the configuration properties of the node. The semantics will become:

1. If msg.filename is present, use that.
2. If msg.filename is not present then use the configuration property of file name
3. If neither msg.filename nor a configuration property for file name are present then that should be considered an error.

ProjectID determination from GCE environment for BigQuery node ...

The BigQuery node needs a projectId. This is specified (today) in the configuration of the node. If Node-RED runs on GCP then we can determine an implied value. This issue asks for the coding/enhancement to support this idea.

Name property not working for all nodes

Each node has a name property where we can set the name of the node. It has been discovered that there is an error and some of the nodes are missing:

        name: { value: "", required: false },

in their HTML files. This defect asks that we check each node and validate that they are all present.

Sentiment node fails with msg.payload neither String nor Buffer

When we pass a string into the sentiment node, it fails with an error of:

msg.payload neither String nor Buffer

Provide an example for the Monitoring node

We have added a new Monitoring node but have not yet provided an example. This issue is for tracking the work to add a new example for the Monitoring node.

Design and implement support for FireStore

FireStore is a NoSQL database. This issue will track the design and development of FireStore nodes.

Nodes not working on Raspberry Pi

When we install the latest Node-RED on a Raspberry Pi and attempt to use the nodes, we get an error:

Error: Unable to detect a Project Id in the current environment. 
To learn more about authentication and Google APIs, visit: 
https://cloud.google.com/docs/authentication/getting-started

Add Translation node

Google provides a translation API to translate text from one language to another. This node will wrapper that capability.

Provide support for EITHER credentials OR a key file for Sentiment

Today, credentials for accessing GCP can be provided by a credentials property. While this is ok, we want an alternate story which is the ability to provide a GCP service account JSON key file. We want to do this with the addition of a new property ... today we have account ... we want to add a new property called keyFilename.

This particular issue is for the Sentiment node.

Problem with GCS Write: Type Error, cannot read property "ending" of undefined

The following was posted by user profDgasca:

Hi,
I try to create a new jpg file from node-red in a raspi, mi npm version is 6.4.1
my configuration is:

msg = {
filename:'gs://bucket/test.jpg',
payload:msg.payload,
contentType:'image/jpeg'
}

where
msg.payload is a buffer

The error is:
TypeError: Cannot read property 'ending' of undefined

It's somthing wrong with my config?
thanks a lot and regrads

@profDgasca

Bullet nesting looks poor in Speech to Text markdown

If we look here, https://github.com/GoogleCloudPlatform/node-red-contrib-google-cloud/blob/HEAD/docs/speech_to_text.md the nested bullets look wrong.

Debug statement left in code

Yesterday, I made a fix to the PubSub in node and left a debug statement in the code. This issue tracks the removal of that debug statement.

Perform decoding for JSON based data in PubSub in

It is not uncommon for the payload of a PubSub message to be a JSON string. Today, we always pass that data on as a Buffer and leave it to the downstream nodes to decode to JSON if needed. This issue asks for an option to decode the payload from a JSON string to an object representation. The selection of whether or not to parse the data will be made as a user configurable option.

writeStream error: Service Unavailable

Hi,

After some hours of bad reading from my side, i finally managed to get something back from the google cloud services.

But i got this as response:
{"_msgid":"7d804f9f.8910f",
"topic":"",
"payload":"home/pi/Pictures/hoi.jpg",
"filename":"gs://[bucketname]/[home/pi/Pictures/hoi.jpg]"}

Google console tells me everything should be up and running so im stuck where to look further now

Provide support for EITHER credentials OR a key file for Logging

Today, credentials for accessing GCP can be provided by a credentials property. While this is ok, we want an alternate story which is the ability to provide a GCP service account JSON key file. We want to do this with the addition of a new property ... today we have account ... we want to add a new property called keyFilename.

This particular issue is for the Logging node.

Docs on firestore not right ...

If we look at the formatted docs for Firestore here:

https://github.com/GoogleCloudPlatform/node-red-contrib-google-cloud/blob/master/docs/firestore.md

They don't look right.

In addition, the primary README.md at the root of the project doesn't list Firestore.

Speech to Text GCP node

GCP exposes a Speech to Text service. This issue is to be used for the tracking of a new Node-RED node that exposes the Speech to Text processing.

Provide an example for Speech to Text

We don't yet have an example of Speech to Text node. This issue is to be used to track the creation of such.

Provide support for EITHER credentials OR a key file for GCS write

Today, credentials for accessing GCP can be provided by a credentials property. While this is ok, we want an alternate story which is the ability to provide a GCP service account JSON key file. We want to do this with the addition of a new property ... today we have account ... we want to add a new property called keyFilename.

This particular issue is for the GCS write node.

Replace npm dependency 'request'

The Get Metadata node had a dependency on the npm package called 'request'.

https://www.npmjs.com/package/request

This has been deprecated.

This issue tracks the replacement which will be 'axios'.

https://www.npmjs.com/package/axios

Provide support for EITHER credentials OR a key file for PubSub in

Today, credentials for accessing GCP can be provided by a credentials property. While this is ok, we want an alternate story which is the ability to provide a GCP service account JSON key file. We want to do this with the addition of a new property ... today we have account ... we want to add a new property called keyFilename.

This particular issue is for the PubSub in node.

Provide support for EITHER credentials OR a key file for Monitoring

Today, credentials for accessing GCP can be provided by a credentials property. While this is ok, we want an alternate story which is the ability to provide a GCP service account JSON key file. We want to do this with the addition of a new property ... today we have account ... we want to add a new property called keyFilename.

This particular issue is for the Monitoring node.

Expose content type as a configuration property

Currently, the content type of a GCS Write is supplied at runtime in the msg.contentType property. This issue requests and enhancement to allow the content Type to be supplied via a configuration property. If both a runtime and a configuration property are supplied, the runtime will be used.

Request: Have gcs-write node notify when it's done writing.

Howdy,

Thanks for creating a great node toolset. One of my teammates is using the gcs-write node. He asserts that the node does not tell us when it's done writing.

Can you alter the node so it notifies us when it's done writing?

I'm happy to answer any questions you may have.

Thank you,

Jonathan L.

Procedure to use pubsub node in rasPi to send data to google cloud

Hello,

I am very new to using google cloud. I am trying to establish communication between node-red (running on rasPi) and google cloud. I am now in a situation that says "Error: 7 PERMISSION_DENIED: User not authorized to perform this action."
I have generated and downloaded the key JSON file and mapped the content in the pubsub node, specified the topic, but no luck yet! Can you please help me sort out this issue?

Thanks in advance.

Concat error with gcs read

The GCS read node was failing to read data throwing an error with a function called concat. Somehow it was felt that we had a function called concat on a buffer object instance but that is plain wrong. The concat method exists of Buffer class.

How to connect google MySQL with node red?

Hi, currently i am developing my final year project regarding running a node red system and database in the cloud. I am using google cloud platform for this current project.
I am wondering is there any way i could connect google MySQL database to the node-red running on my virtual machine? I tried using the MySQL pallette but it keep on giving me

"Error: ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MariaDB client"

"Database not connected"

"Error: Pool is closed."

Is there any way i could connect them? I tried google for any other solutions but can't find any

GCS Read would fail if no filename supplied in configuration at coding level

The GCS Read node expects a configuration property called filename. This works 99.9% of the time as it has a default value of "" when created in the editor. However, since this field is optional it would be legal to not supply a value at all and hence the property would simply not exist (as opposed to existing and being the empty string). This caused breakage in the logic. This issue asks for the resolution of that to handle the complete absence of the property.

docs/bigquery.md contains bad image link

Within the docs/bigquery.md file there is a line which reads:

![BigQuery Node](images/nigquery_node.png)

This contains a typo and should be:

![BigQuery Node](images/bigquery_node.png)

Provide support for EITHER credentials OR a key file for Vision

Today, credentials for accessing GCP can be provided by a credentials property. While this is ok, we want an alternate story which is the ability to provide a GCP service account JSON key file. We want to do this with the addition of a new property ... today we have account ... we want to add a new property called keyFilename.

This particular issue is for the Vision node.

Help requested with iot core

In Node-Red, what would be easiest way to send commands or configurations to IoT Devices?

There is node.js example in https://cloud.google.com/iot/docs/how-tos/commands -> API -> Node.js

But it seems to take quite few steps to take it in use if running .js code in Node-Red which is running in Docker image.

Node-red with docker on raspberry Pi

I have installed Node-Red with docker in a RPI and I'm trying to connect to GCP, so far unsuccessfully.

I get this error in the debug messages:

Error: Unable to detect a Project Id in the current environment.
To learn more about authentication and Google APIs, visit:
https://cloud.google.com/docs/authentication/getting-started

It seems like it has to do with the environment variable GOOGLE_CLOUD_PROJECT. Reading the installation steps, I get stuck in this part as it is not started through the systemctl daemon:

"...Note: For Raspberry Pi users - There is the option of starting Node-RED automatically through the Linux systemctl daemon. This system does not use global environment variables and thus you must explicitly define the variable to use.

Edit the file /lib/systemd/system/nodered.service

and add a line which reads:

Environment="GOOGLE_CLOUD_PROJECT=" ..."

I have added the JSON key message in the credentials but it doesn't seem to do the trick yet.

Help would be much appreciated. Thanks

Add attributes to the output of the PubSub-In node

Today the data property of the PubSubMessage is mapped to msg.payload.
https://github.com/GoogleCloudPlatform/node-red-contrib-google-cloud/blob/master/pubsub-in.js#L77-L79

Would be good to also output the attributes attached to a PubSubMessage to msg.attributes or similar.

Something like this (Note, I have never developed any Node-RED nodes though).

const msg = {
   payload: message.data,
   attributes: message.attributes
};

Made this small change locally and it works.

Provide node to other langagues

Can't change the default language en-US to another that is already supported by Google API. Only changing at .json and .js files at /node_modules from en-US to es-ES can't help...it just recognizes english. Creating a field with language option would be very helpfull for the community

Firestore probrem mode:set, update, delete, get

I have installed Firestore in Node-RED.
But except mode:query, unsuccessfully.

The path to the document see below.
msg.payload.path = /collection/document

ex
if (mode == 'set' || mode == 'update' || mode == 'delete' || mode == 'get') {
const document = firestore.doc(msg.payload.path);
}

Help would be much appreciated. Thanks

Add support for Application Default Credentials

It is a given that calls to GCP services must be authenticated. Currently we provide the ability to provide credentials (JSON) as either a file or a Node-RED secret ... but there is a 3rd way ... we simply specify nothing and the environment gives us a default identity. This is true if we are running on a compute engine, GKE or ... for my immediate purposes ... Cloud Run. This issue will track the ability to provide NO definition for security credentials resulting in the defaults being used.

Provide support for EITHER credentials OR a key file for GCS read

Today, credentials for accessing GCP can be provided by a credentials property. While this is ok, we want an alternate story which is the ability to provide a GCP service account JSON key file. We want to do this with the addition of a new property ... today we have account ... we want to add a new property called keyFilename.

This particular issue is for the GCS read node.

Add an output to the Publish Node

The current GCP Publish Node has no output. That means that we can't know when a publish has completed. This issue asks for an Output terminal that will be fired when the publish has completed.

pubsub node bug

There is minor bug in pubsub node. If using "Assume JSON" option and the received pubsub message is not valid JSON then there will be uncaughtException and whole node red will shut down. ref https://nodered.org/docs/user-guide/handling-errors

Work around is simple: Not to use "Assume JSON" but instead parse JSON on separate node.

Add a DLP node

Add a node that encapsulates Data Loss Prevention API calls.

problems with pubsub

I tested https://github.com/GoogleCloudPlatform/node-red-contrib-google-cloud/blob/master/examples/PubSub.json and got it working correctly with GCP compute engine which was running node red container.

When I tried to do processing to received pubsub message I faced problems. The whole container crashed with following error:
28 Feb 13:57:19 - [red] Uncaught Exception:
28 Feb 13:57:19 - Error [ERR_HTTP2_NO_SOCKET_MANIPULATION]: HTTP/2 sockets should not be directly manipulated (e.g. read and written)
at Object.get (internal/http2/core.js:814:15)
at _clone (/usr/src/node-red/node_modules/clone/clone.js:162:31)
at _clone (/usr/src/node-red/node_modules/clone/clone.js:175:25)
at _clone (/usr/src/node-red/node_modules/clone/clone.js:162:18)
at _clone (/usr/src/node-red/node_modules/clone/clone.js:162:18)
at _clone (/usr/src/node-red/node_modules/clone/clone.js:162:18)
at _clone (/usr/src/node-red/node_modules/clone/clone.js:162:18)
at _clone (/usr/src/node-red/node_modules/clone/clone.js:162:18)
at /usr/src/node-red/node_modules/clone/clone.js:141:24
at Map.forEach ()
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /usr/src/node-red/.npm/_logs/2020-02-28T13_57_19_714Z-debug.log

The flow what I used attached. It is based on the above mentioned example.
flow.txt

language-sentiment to other languagues

Hello again!

I'm here to ask for the same enhancement you did past weeks on speech node, so I can use the language-sentiment node in other language as I could not change de default en-US.

Thank you again!

Provide support for EITHER credentials OR a key file for PubSub out

Today, credentials for accessing GCP can be provided by a credentials property. While this is ok, we want an alternate story which is the ability to provide a GCP service account JSON key file. We want to do this with the addition of a new property ... today we have account ... we want to add a new property called keyFilename.

This particular issue is for the PubSub out node.

Design and implement support for a BigQuery node

BigQuery is a Google Data Warehouse service. The purpose of this issue is to track the creation of a new Node-RED node that provides BQ support.

Unable to authenticate

Hi,

I followed instructions from https://cloud.google.com/blog/products/application-development/using-node-red-with-google-cloud and created Compute Engine with node red container.

Then tested this example https://github.com/GoogleCloudPlatform/node-red-contrib-google-cloud/blob/master/examples/PubSub.json

I created pubsub topic and subscription for "test-topic" and added "owner" role for the service account of the VM where node red is running.

When deploying the node-red flow, both pubsub nodes in the flow gives error "Error: 7 PERMISSION_DENIED: Request had insufficient authentication scopes."

Provide support for EITHER credentials OR a key file for Tasks

Today, credentials for accessing GCP can be provided by a credentials property. While this is ok, we want an alternate story which is the ability to provide a GCP service account JSON key file. We want to do this with the addition of a new property ... today we have account ... we want to add a new property called keyFilename.

This particular issue is for the Tasks node.

Specify a file name in the configuration for GCS Write

Currently, when we use the GCS Write node, we expect to find a file name in the msg.filename property at runtime. While this is great, we can do more. This issue suggests a new feature that provides an optional file name in the configuration properties for the node. The suggested semantics are:

• If msg.filename exists ... use it
• if msg.filename does NOT exist then use the configured filename supplied by the flow designer
• If neither msg.filename nor a configured filename exist, then that would be an error.

Race condition in PubSub out when instance newly created

Imagine we have a flow which contains a PubSub out node. The implementation of this node calls GCP to get the Topic which is then used to send messages. However, there is the possibility that the node will have received requests to publish messages BEFORE the PubSub node is ready.

PubSub in messages are read-only but aren't being treated as such.

With the PubSub In node, we are being triggered when a new message is published to a topic. We are being passed a Message object and going forth from there. We are currently passing the receives message as the Node-RED msg.payload which means that it is fair game for downstream nodes to "fiddle" with the content. That must NOT be allowed. Our initial thinking is that we will change the semantics of the node.

When a new message is received, we will place that message at:

msg.message and document it as being read-only.

We will place the data payload of the received message at msg.payload which should be fine as it will be a Buffer and immutable.

This will be a semantic change on the node.