kubeedge/mappers-go

KubeEdge Device Mappers written in go

English | 简体中文

mappers-go

KubeEdge Device Mappers written in go, it works for KubeEdge 1.4 version and later.

Construction method

If you want to connect your edge device to kubeedge, you can create and write mapper to complete it. Now, two methods are provided, the basic mappers and mapper-sdk-go.

mappers

Mappers implement modbus, bluetooth, onvif and opcua. If you want to connect devices to kubeedge of these protocols, you can find them directly in mappers.

mapper-go-sdk

Mapper-sdk-go is a basic framework written in go. Based on this framework, developers can more easily implement a new mapper. Mapper-sdk has realized the connection to kubeedge, provides data conversion, and manages the basic properties and status of devices, etc. Basic capabilities and abstract definition of the driver interface. Developers only need to implement the customized protocol driver interface of the corresponding device to realize the function of mapper. You can get more information and details in mapper-sdk-go

mappers-go's Issues

Smart meter protocols

It is possible to add mappers to Smart meter protocols DL/T 645-1997 & DL/T 645-2007?

Modbus&opcua device example configuration issue.

The value type of twin could not be written as "integer" but "int".

opcua Datatype does not match

When I use int type data to write to the server, the server shows that the data type does not match .But when I use the string type to operate, this problem is solved .

Apply informer-operator design pattern to implement the drivers

The informer-operator design pattern is popular in k8s, especially for the event driven component such as deploy-controller.

With a deep insight, the mapper is an event driven controller indeed. The only different from the deploy controller is the event source. Event of the former comes from the api server while event of the mapper comes from the change of disk file(mount configmap as a local file) and MQTT, I think it is worth to make the two event sources(Or anything that the event comes from) as an Informer implementation, so the driver developer can write their mapper as what they do in operator development, it is more friendly for them.

Thanks!

Umbrella Issue for Developing Plugin of Mapper Based on DMI

Hello everyone!

We have proposed a new device management framework for KubeEdge, Device Management Interface, also known as DMI. Based on the KubeEdge DMI framework, a Mapper plugin corresponding to the edge smart device protocol is developed to map edge devices to KubeEdge as device twins and manage the lifecycle of edge smart devices in cloud native mode.

We have also developed a mapper for modbus with DMI as an example. Developers are welcome to develop mappers of all kind of protocols based on DMI and propose PR to this mappers-go repository.

And we also want to devevlop mappers with other languages because DMI depends on gRPC which is not language-bound. Besides Golang, C or Python is also what we plan to develop with.

If you have any questions about developing mappers with DMI, please feel free to leave a message here. Look forward to your contribution to KubeEdge. Thanks!

mapper映射器

Can Kubeedge currently map a machine node through the current mapper

bug: mappers-sdk's Parse can't parse customizedProtocol correctly

When users use mappers-sdk to generate a customized mapper from template, Parse will check protocol name:

if instance.PProtocol.Protocol != "Template" {
	continue
}

This is checking protocols.protocol in deviceProfile.json.
However, the ConfigMap associated with device-instance didn't contain any protocol named "Template". The "Template" appears in protocols.protocolConfig.protocolName
eg:

"protocols": [{
	"name": "customized-protocol-xxxx",
	"protocol": "customized-protocol",
	"protocolConfig": {
		"protocolName": "Template"
	},
	"protocolCommonConfig": null
}]

So, there is a bug, the checking code mentioned above will always exec continue!
There are two ways to fix:

1. modify CRD and device-instance.yaml.
2. modify the checking code.

The second way is easier

failed to build modbus image

Ref kubeedge/kubeedge#2512

/assign @sailorvii

opcua 写值失败的问题

mappers\opcua\driver\client.go
Set的时候所写的opcua节点的属性必须是String类型的才可以成功。
如果opcua节点的是Int32的，此时Set会失败。
参考：gopcua/opcua#310

func (c *OPCUAClient) Set(nodeID string, value string) (results string, err error) {
    ///...

	v, err := ua.NewVariant(value) //type of value is string
	if err != nil {
		klog.Errorf("invalid value: %v", err)
		return "", errors.New("Invalid value")
	}

	req := &ua.WriteRequest{
		NodesToWrite: []*ua.WriteValue{  // WriteValue failed if the type of node is not string.
			{
				NodeID:      id,
				AttributeID: ua.AttributeIDValue,
				Value: &ua.DataValue{
					EncodingMask: ua.DataValueValue,
					Value:        v,
				},
			},
		},
	}

	resp, err := c.Client.Write(req)

此处想写一个Int32的Node必须这样写：

func (c *OPCUAClient) Set(nodeID string, value string) (results string, err error) {
   ....
       // convert string to int32
	value64, err := strconv.ParseInt(value, 10, 32)
	if err != nil {
		klog.Errorf("value convert failed : %v", err)
	}
        var val32 int32
	val32 = int32(value64)
	//v, err := ua.NewVariant(value)
	v, err := ua.NewVariant(val32)  
      ...

如果运行gopcua自带的demo，写一个Int32型变量，也是一样的问题

go run examples/write/write.go -endpoint opc.tcp://192.168.1.228:4840  -node 'ns=1;s=start' -value 0
The value supplied for the attribute is not of the same type as the attribute's value. StatusBadTypeMismatch (0x80740000)

DMI mapper of modbus cannot reconnect to device automatically

If a device disconnect to the DMI mapper of modbus, the mapper cannot reconnect to the device automatically. It only works after the mapper is rebooted. We should reconnect to the device automatically if the mapper fails to write data to the device.

After the modbus device connects to the mapper, the container of the mapper fails.

The Modbus Mapper is running normally. After the device is delivered, the container of the mapper fails.

View the mapper log. The command output is "panic: runtime error: invalid memory address or nil pointer dereference".

Locate the problem. It was found that the payload in the Run() function, located in the mappers/modbus-dmi/device/twindata.go file, did not add the "Expected" data section. This results in an error when accessing msg.Twin.Expected when executing the parse.ConvMsgTwinToGrpc(msg.Twin) function.
Is this a bug in the process of design and implementation? If so, please leave bugfix matters to me.

Add e2e and ut for the repo

a proposal for mapper-sdk

Hi guys, we have make a proposal for mapper-sdk, which provides a more efficient method for accessing a new protocol.

What would you like to be added/modified:

1. Add mapper-sdk.
2. Provide a RESTful API.

Why we need it:

1. Now there are a lot of reusable parts in mapper's code. Mapper-sdk encapsulates these parts into functions to help developers reduce such meaningless things.
2. Mqtt does not support the transmission of large packets of data. The RESTful API can alleviate this problem.

The preliminary plan can be viewed in the following documents :)
https://docs.google.com/document/d/1KrsFfPP8KZ33al3iB4g8hNWxi4sKtOWkkizLFyQAHSE/edit?usp=sharing

The preliminary code can be found in the following repository :)
https://github.com/QiQi-OvO/mappers-go

Welcome to discuss more details about this proposal and give some suggestions，thanks.

创建个模板各种执行中断 没人维护的吗

Modbus collects data successfully, twins reported data does not change

environment:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
edge-node-1 Ready agent,edge 7d1h v1.19.3-kubeedge-v1.5.0
edge-node-2 Ready agent,edge 5d5h v1.19.3-kubeedge-v1.5.0
k8s-master Ready control-plane,master 7d2h v1.20.0
node-2 Ready 6d v1.20.0
raspberrypi Ready agent,edge 46h v1.19.3-kubeedge-v1.5.0

kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:51:19Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

docker version
Client: Docker Engine - Community
Version: 20.10.0
API version: 1.41
Go version: go1.13.15
Git commit: 7287ab3
Built: Tue Dec 8 18:54:00 2020
OS/Arch: linux/amd64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.0

Question 1:
The data format of modbus go read data and print is as follows:
I0127 08:10:48.180520 1 client.go:150] Get result: [0 44]
https://github.com/beyondyinjl2/bug-images/blob/main/modbus/1611738342465.jpg
https://github.com/beyondyinjl2/bug-images/blob/main/modbus/1611737588077.jpg
Do you take the second place of the data?
vim pkg/modbus/device/twindata.go
td.Results[1]
https://github.com/beyondyinjl2/bug-images/blob/main/modbus/1611738635412.jpg

Question 2:
twins reported data does not change?
There is data in mqtt as shown in the figure:
https://github.com/beyondyinjl2/bug-images/blob/main/modbus/1611737459271.jpg
https://github.com/beyondyinjl2/bug-images/blob/main/modbus/1611737509080.jpg

twins reported data does not change as shown in the figure:
kubectl get device modbustcp-device -oyaml -w
https://github.com/beyondyinjl2/bug-images/blob/main/modbus/1611737626760.jpg

opcua field of opcua config.yaml not used, can be removed

opcua field of opcua config.yaml not used, can be removed

mqtt:
  server: tcp://127.0.0.1:1883
  username: ""
  password: ""
  certificationFile: ""
  privateKeyFile: ""
configmap: /opt/kubeedge/deviceProfile.json
opcua:                                              ----------------------not used, can be deleted
  username: ""
  password: ""
  certificationFile: ""
  privateKeyFile: ""
  remoteCertificationFile: ""

Modbus go mapper error

sudo docker build -t modbusmapper:v1.0 ./pkg/modbus

Sending build context to Docker daemon 52.74kB
Step 1/6 : FROM ubuntu:16.04
---> 9499db781771
Step 2/6 : RUN mkdir -p kubeedge
---> Running in b680bf86b7e1
Removing intermediate container b680bf86b7e1
---> e88ad3020b46
Step 3/6 : COPY ./modbus kubeedge/
COPY failed: stat /var/lib/docker/tmp/docker-builder170465191/modbus: no such file or directory

not modbus dir

GB/T 28181 mapper Support

hi,all
I saw that there is a GB/T 28181 mapper plan for 2021 H2 in the kubeedge.io Road Map, but I can’t find it in mappers-go. onvif and opu-ca are ready. Is the GB/T 28181 mapper plan delayed?
In addition, the GB/T 28181 mapper should be a complicated project. Based on my experience in the security industry, if it does not involve the processing of video streams from security manufacturers, it will be easier to be a mapper for device control.

add mqtt device mapper

This is an implementation of mapper for mqtt protocol. The aim is to create an application through which users can operate mqtt device and read mqtt device data.

the part of inittwin in ble mapper

now i have a ble device , the twin propertyName is on-off , as it's name ,it control the sensor on-off ,the question is that it's property is write,so i can't read it's value ,but the twin has the desire and report value,so how can i get the report value?
in the code about the func inittwin ,after the func setVisitor finish,it begin to judge whether the property is notify or read ,and then send the report value.but the property is write ,so i can't get the report value.
the code is from device.go

if (c.Property&ble.CharNotify) != 0 && c.CCCD != nil {
	wg.Add(1)
	go func() {
		if err := twinData.BleClient.Client.Subscribe(c, false, twinData.notificationHandler()); err != nil {
			klog.Errorf("Subscribe error: %v", err)
			wg.Done()
		}
	}()
} else if (c.Property & ble.CharRead) != 0 { // // read data actively
        timer := mappercommon.Timer{Function: twinData.Run, Duration: collectCycle, Times: 0}
	wg.Add(1)
	go func() {
		defer wg.Done()
		timer.Start()
	}()
}

启动后没有正确解析deviceProfile.json

• 版本
  kubeedge: 1.8.2
  mapper-go: 分支release-1.8

• 问题描述：
  启动后没有将modbus协议的点位解析出来，导致后面设置desire的代码永远都不会执行
  `

mappers-go\mappers\modbus\configmap\parse.go

for j = 0; j < len(deviceProfile.Protocols); j++ {
if instance.ProtocolName == deviceProfile.Protocols[j].Name {
instance.PProtocol = deviceProfile.Protocols[j]
break
}
}
if j == len(deviceProfile.Protocols) {
err = errors.New("Protocol not found")
return err
}

if instance.PProtocol.Protocol != "modbus" {
continue
}
`
instance.ProtocolName 这里的 ProtocolName 其实在deviceProfile.json中并不存在
在云端发布设备实例之后，在congfigmap中的deviceProfile.json里面，只有一个叫Protocol的字段来标识协议名称

why mapper Twin structure has PropertyVisitor field

why mapper Twin structure has PropertyVisitor field, on the cloud side, this Twin structure only has three fields except PropertyVisitor

type Twin struct {
	PropertyName string `json:"propertyName,omitempty"`
	PVisitor     *PropertyVisitor
	Desired      DesiredData  `json:"desired,omitempty"`
	Reported     ReportedData `json:"reported,omitempty"`
}

connect to modbus device failed

hi all,
mapper connects to modbus devices in modbus-TCP mode,can connect to a device,But the HoldRegister data type read failed.
The questions are as follows:
"read tcp ip:port->device IP:502 i/o timeout",ping device ip address is OK,the ip address of the telnet device and port 502 are normal.The error location is twindata.go line 127

Create mapper template

Create a template for a customized mapper to ease the mapper development.

the part of inittwin in ble mapper

now i have a ble device , the twin propertyName is on-off , as it's name ,it control the sensor on-off ,the question is that it's property is write,so i can't read it's value ,but the twin has the desire and report value,so how can i get the report value?
in the code about the func inittwin ,after the func setVisitor finish,it begin to judge whether the property is notify or read ,and then send the report value.but the property is write ,so i can't get the report value.
the code is from device.go

if (c.Property&ble.CharNotify) != 0 && c.CCCD != nil {
	wg.Add(1)
	go func() {
		if err := twinData.BleClient.Client.Subscribe(c, false, twinData.notificationHandler()); err != nil {
			klog.Errorf("Subscribe error: %v", err)
			wg.Done()
		}
	}()
} else if (c.Property & ble.CharRead) != 0 { // // read data actively
        timer := mappercommon.Timer{Function: twinData.Run, Duration: collectCycle, Times: 0}
	wg.Add(1)
	go func() {
		defer wg.Done()
		timer.Start()
	}()
}

a proposal for gige-sdk

Hi, guys. We have make a proposal for gige-sdk, which is a GigE mapper based on mapper-sdk-go and supports GigE Vision protocol cameras access.

More details about gige-sdk: gige-sdk README.md

We can discuss more details about this proposal and welcome to make suggestion for gige-sdk, thanks.

What would you like to be added/modified:
We want to use the generic camera protocol genicam protocol as the driver of the GigE mapper to access cameras from different manufacturers, so there is no need to adapt the driver layer for cameras from different manufacturers and reduce customized development. At the same time, we want to develop GigE mapper based on mapper-sdk-go and encourage other people to use this new mapper-sdk to help us develop a new mapper more easily.

Why we need it:
At present, KubeEdge mapper does not have a mapper suitable for GigE vision cameras access. At the same time, there is no good solution for cameras access from different manufacturers, which can not meet the needs of users for mapper.

Unable to build bluetooth mapper image using the new mapper make template

What happened and what you expected to happen:
Since version 1.7, the new mapper template to build bluetooth image is not working.

How to reproduce it (as minimally and precisely as possible):
Run make mapper bluetoothmapper_image b ARM=true

Anything else we need to know?:
Error log:
bluetoothmapper_image b
--make mapper bluetoothmapper_image b
downloading dependencies for mapper bluetoothmapper_image...
tidying
vending
...done
fmt and linting mapper bluetoothmapper_image...
WARN [runner] The linter 'golint' is deprecated (since v1.41.0) due to: The repository of the linter has been archived by the owner. Replaced by revive.
driver/client.go:45:3: undeclared name: klog (typecheck)
klog.Errorf("New device error: %v", err)
^
driver/client.go:62:5: undeclared name: klog (typecheck)
klog.Errorf("Write characteristic error %v", err)
^
driver/client.go:26:2: "k8s.io/klog/v2" imported but not used (typecheck)
"k8s.io/klog/v2"
^
device/device.go:45:3: undeclared name: klog (typecheck)
klog.V(1).Info("Visit readonly characteristicUUID: ", visitorConfig.CharacteristicUUID)
^
device/device.go:28:2: "k8s.io/klog/v2" imported but not used (typecheck)
"k8s.io/klog/v2"
^
device/devicestatus.go:20:2: "k8s.io/klog/v2" imported but not used (typecheck)
"k8s.io/klog/v2"
^
../../../../../../snap/go/7736/src/runtime/cgo/cgo.go:34:8: could not import C (cgo preprocessing failed) (typecheck)
import "C"

Environment: Ubuntu 20.10

opcua runtime fail

Opcua agreement mapper run time card in the Subscribe topic: $hw/events/device/lamp/twin/update/delta, deviceProfile. Json what need to modify the configuration file