cloudwego/dynamicgo

Dynamically and efficiently operate RPC data for Go

Dynamic-Go

Dynamically operating data for Go. Aiming at reducing serialization/deserializtion process thus it can be fast as much as possible.

Introduction

Dynamic-Go for Thrift protocol: introduction.md.

Dynamic-Go for Protobuf protocol: introduction.md

Usage

thrift

Thrift IDL parser and message operators. It can parse thrift IDL in runtime and handle thrift data in generic way. DOC

thrift/generic

Reflection APIs to search, modify, deserialize, serialize thrift value with or without runtime type descriptor. DOC

thrift/base

The meta data about message transportation, including caller, address, log-id, etc. It is mainly used for conv (protocol convertion) modules.

thrift/annotation

Built-in implementation of thrid-party annotations, see thrift_idl_annotation_standards. They are mainly used for conv (protocol convertion) modules.

proto

Protobuf IDL parser and message operators. It can parse protobuf IDL in runtime and handle protobuf data in generic way. DOC

proto/generic

Reflection APIs to search, modify, deserialize, serialize protobuf value with or without runtime descriptor. DOC

proto/protowire

Protobuf data encode and decode APIs. It parses and formats the low-level raw wire encoding. It is modified from Protobuf official code encoding/protowire. DOC

proto/binary

BinaryProtocol tool for Protobuf Protocol. It can read, wirte and skip fields directly on binary data of protobuf message. DOC

http

Http request/response wrapper interfaces. They are mainly used to pass http values on http<>thrift conversion. DOC

conv

Protocol convertors. Based on reflecting ability of thrift, json and protobuf modules, it can convert message from one protocol into another.

conv/j2t

Convert JSON value or JSON-body HTTP request into thrift message. DOC

conv/t2j

Convert thrift message to JSON value or JSON-body HTTP response. DOC

conv/j2p

Convert JSON value into protobuf message. DOC

conv/p2j

Convert protobuf message into JSON value. DOC

dynamicgo's Issues

UnwrapBinaryMessage return EOF error when biz resp is nil

Describe the bug

thrift.UnwrapBinaryMessage could not unwrap empty biz buf

To Reproduce

func TestBinaryUnwrap(t *testing.T) {
	b := BinaryProtocol{
		Buf:  nil,
		Read: 0,
	}
	var i32 int32 = -2147418110
	b.WriteI32(i32)
	b.WriteString("DummyNew")
	b.WriteI32(1)
	b.WriteByte(byte(STOP))
	_, _, _, _, _, err := UnwrapBinaryMessage(b.Buf)

	t.Errorf("UnwrapBinaryMessage() error = %v", err)

}

Steps to reproduce the behavior:

run the test case

Expected behavior

err is nil and body is empty slice bytes

Screenshots

If applicable, add screenshots to help explain your problem.

Version:

v0.3.0

Environment:

Additional context

Add any other context about the problem here.

R_X86_64_PC32 against undefined symbol `github.com/cloudwego/dynamicgo/internal/native/avx2.__quote'

编译出错，
2023/10/24 22:11:24 go: downloading github.com/cloudwego/dynamicgo v0.1.3

2023/10/24 22:11:31 + export GO111MODULE=off
2023/10/24 22:11:31 + GO111MODULE=off
2023/10/24 22:11:31 + myfile=./vendor/golang.org/x/net/trace/trace.go
2023/10/24 22:11:31 + startLine=120
2023/10/24 22:11:31 + endLine=132
2023/10/24 22:11:31 + [[ linux-gnu =~ ^darwin ]]
2023/10/24 22:11:31 + '[' -f ./vendor/golang.org/x/net/trace/trace.go ']'
2023/10/24 22:11:31 + sed -i '120,132s/^////' ./vendor/golang.org/x/net/trace/trace.go
2023/10/24 22:11:31 + startLine=74
2023/10/24 22:11:31 + endLine=74
2023/10/24 22:11:31 + [[ linux-gnu =~ ^darwin ]]
2023/10/24 22:11:31 + '[' -f ./vendor/golang.org/x/net/trace/trace.go ']'
2023/10/24 22:11:31 + sed -i '74,74s/^////' ./vendor/golang.org/x/net/trace/trace.go
2023/10/24 22:11:31 + mkdir -p output
2023/10/24 22:11:31 + go build -buildmode=plugin -o ./output/stress.so main.go
2023/10/24 22:13:34 # command-line-arguments
2023/10/24 22:13:34 /usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
2023/10/24 22:13:34 /usr/bin/ld: /tmp/go-link-419334024/go.o: relocation R_X86_64_PC32 against undefined symbol `github.com/cloudwego/dynamicgo/internal/native/avx2.__quote' can not be used when making a shared object; recompile with -fPIC
2023/10/24 22:13:34 /usr/bin/ld: final link failed: Bad value
2023/10/24 22:13:34 collect2: error: ld returned 1 exit status

incorrect spell of Descriptor

NewDescritorXXX in thrift/idl.go everywhere should be NewDescriptorXXX :)

protobuf 增加从byte数组中读取指定数量field的函数

protobuf的编解码属于属于位置不可移动类型. 如果能确定编码的时候，某几个字段在生成的字节数组，那么，对于转发或者代理之类的网关，就可以不需要解码整个消息，之需要解码前面几个字段。

生成器生成protobuf代码的时候, 固定写入字段的顺序(编码)，这是可以固定的。另外一个是, 在需要代理转发的地方，根据二进制直接读取前面需要的字段. 目前，dynamicgo中有comsumefixed64这样的方法可以读取用一个字段，是否可以增加，连续读取几个字段的方法呢

对thrift map类型，支持添加新的key-value对

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

[conv/j2p] not support unknown-fields of JSON

dynamicgo/conv/j2p/decode.go

Line 420 in 497e7b3

curDesc = *rootDesc.ByJSONName(key)

this code may panic if parsed JSON key is not defined in IDL. need repair it.

proto/generic Interface 方法增加对 ENUM类型的支持

在使用过程中发现，尝试对二进制数据解码后并转化为 map 类型数据的时候，不支持 ENUM 类型

当初是因为设计上的考虑没有支持 ENUM 么？

是否能够增加如下支持：

Proposal: support high-performance generic-call SDK for `protobuf` protocol

Background

Like Thrift, protobuf also needs a easy-to-use and high-performance generic-calling system in practice.

Goals

• Implement a generic calling SDK based on protobuf protocol, supporting both server and client
• Meet primary needs for generic calling, including routing (optional)、type-describing、protocol-conversion、dynamic-reflection (searching and serialization/deserialization)
• High performance and wide compatiblity
• (optional) Be compatible with dynamicgo's thrift/generic API in the aspect of dynamic reflection.

Alternatives

• Kitex's protobuf generic-calling SDK is applicable only on HTTP(json)-conversion mode, and performance is not well enough.
• Offcial SDK support dynamic reflection and json<>protobuf conversion both, but alse has performance problem and not compatible with DSL tools like GraphGL well enough.

Desgin Advices

• For type-describing, we could use official implementation, considering the compatibility. But this implementation will have some impact on performance (not sure) ?
• For protocol-conversion, we can implement it using the way dynamicgo/conv does —— converting in place and sequentially, without any intermediator
• For dynamic-reflection, we can still split it into two APIs -- searching (based on skipping) and serializing/deserializing (based on DOM). For searching, one important thing is the speed of skipping algorithm —— I'm afraid that varint encoding has big impact on it. For full-load serializing/deserializing, I want to keep the same memory model with Thrift's DOM, for future conversion of these two —— in application on DSL like GraphGL.