bytedance / go-tagexpr Goto Github PK
View Code? Open in Web Editor NEWAn interesting go struct tag expression syntax for field validation, etc.
License: Apache License 2.0
An interesting go struct tag expression syntax for field validation, etc.
License: Apache License 2.0
无tag字段先绑定cookie再绑定header,在绑定cookie时,如果err == nil就认为绑定成功,但是err == nil 的条件很松,导致绑定到cookie就会终止,不会去绑定header。能不能提高下err == nil的标准或者先bind header再bind cookie呢?
it works on ctx.Bind(req)
and works again on validator.Validate(&req)
. the message is so boring.
var debugSwitch = goutil.IsGoTest()
func printf(format string, a ...interface{}) {
if debugSwitch {
fmt.Fprintf(os.Stderr, format, a...)
}
}
how to modify the debugSwitch
?
only one test case you could have the following lines
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── >
└── 0
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── >
└── 0
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── >
└── 0
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >
└── 0
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── >
└── 0
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── !=
└──
└── ()
┌── ()$
└── >=
└── 0
└── ()
┌── ()$
└── >
└── 0
└── ()
┌── ()$
└── !=
└──
reproduce as code below:
type MyStruct struct {
Data map[string]interface{}
}
func main() {
ms := MyStruct{
Data: map[string]interface{}{
"a": "111",
"b": nil,
},
}
vd := validator.New("vd")
err := vd.Validate(ms)
fmt.Println(err)
}
stack info:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x2 pc=0x101177a]
goroutine 1 [running]:
reflect.maplen(0x2, 0x15)
/Users/qujianping/local/go/src/runtime/map.go:1371 +0xa
reflect.Value.MapKeys(0x13b7ba0, 0xc0000aff20, 0x195, 0x14262ad, 0x1, 0x1426442)
/Users/qujianping/local/go/src/reflect/value.go:1136 +0x208
simple/vendor/github.com/bytedance/go-tagexpr.(*TagExpr).Range(0xc00033c0c0, 0xc0002ceb40, 0x5819e00, 0x0)
/Users/qujianping/local/src/simple/vendor/github.com/bytedance/go-tagexpr/tagexpr.go:792 +0x95c
simple/vendor/github.com/bytedance/go-tagexpr/validator.(*Validator).Validate.func1(0xc00033c0c0, 0x0, 0x0, 0x15e92e0, 0x13cefc0)
/Users/qujianping/local/src/simple/vendor/github.com/bytedance/go-tagexpr/validator/validator.go:78 +0x18c
simple/vendor/github.com/bytedance/go-tagexpr.(*VM).subRunAll(0xc0000aff80, 0x0, 0x0, 0x0, 0x13cefc0, 0xc0000aff20, 0x19, 0xc000163ed8, 0x80, 0x13c73a0)
/Users/qujianping/local/src/simple/vendor/github.com/bytedance/go-tagexpr/tagexpr.go:160 +0xb46
simple/vendor/github.com/bytedance/go-tagexpr.(*VM).RunAny(0xc0000aff80, 0x13cefc0, 0xc0000aff20, 0xc000163ed8, 0xc000163ec8, 0x100d509)
/Users/qujianping/local/src/simple/vendor/github.com/bytedance/go-tagexpr/tagexpr.go:144 +0x82
simple/vendor/github.com/bytedance/go-tagexpr/validator.(*Validator).Validate(0xc000163f68, 0x13cefc0, 0xc0000aff20, 0x0, 0x0, 0x0, 0xc000163f67, 0x1)
/Users/qujianping/local/src/simple/vendor/github.com/bytedance/go-tagexpr/validator/validator.go:70 +0x197
main.main()
/Users/qujianping/local/src/simple/main.go:21 +0x1b0
Test case
import (
"fmt"
vd "github.com/bytedance/go-tagexpr/validator"
"github.com/shopspring/decimal"
)
func main() {
p := struct {
Price decimal.Decimal `vd:"$>0"`
}{
Price: decimal.NewFromFloat(1.00),
}
fmt.Println(vd.Validate(p))
}
would result
invalid parameter: Price
header在业务上有归一化的逻辑,但是在库里没有做归一化,这里希望能支持下。
content-type:application/json时,执行(b *Binding) prebindBody,不会执行注册的unmarsh方法(binding.MustRegTypeUnmarshal)。time.Time非rfc3339格式解析出错就结束解析了。能否time.Time绑定json格式也支持binding.MustRegTypeUnmarshal?
The binding/README.md
say:
If no position is tagged, try bind parameters from the body when the request has body,
otherwise try bind from the URL query
But after I modified the TestAuto, add "c": "c-from-jsonbody"
into NewJSONBody, the case still pass.
go-tagexpr/binding/bind_test.go
Line 664 in 8355f4a
So docs or code, which is wrong?
Hello, we use the go-tagexpr to validate configuration and when Enable is true, check the other fields in struct, for example:
type Config struct {
Enable bool
Servers RelayServerArr `vd:"!(Enable)$ || len($)>0"`
SubStreamMode int `vd:"!(Enable)$ || ($>0 && $<4))"`
}
This looks a bit verbose and the statements in the tag are longer.
Is it possible to add something like stop
? When stop($)
is true, stop validating the other fields after this field.
Thanks
type Config struct {
Enable bool `vd:"stop(!$)"`
Servers RelayServerArr `vd:"len($) > 0"`
SubStreamMode int `vd:"$>0 && $<4"`
}
对自定义枚举值想调用自己的validate method
比如
type Status string
func (s Status) Validate() error { ... }
type Entity struct {
Status Status `vd:"evalfn($)"`
}
type ValidateFn interface {
Validate() error
}
func useValidateFn(args ...interface{}) error {
target, ok := args[0].(ValidateFn)
if !ok {
return fmt.Errorf("evalfn need struct implement ValidateFn interface")
}
return target.Validate()
}
这里args[0]的值类型会是string
go1.16版本下
tagexpr.go 引用的 "github.com/henrylee2cn/goutil" 有问题
goutil 这个东西报错
use of internal package github.com/andeya/goutil/internal/ameda not allowed
在七月份的时候这个goutil是没有问题的 今天导入tagexpr的时候把这个goutil导进来就有问题了 不靠谱的包
我把demo copy到本地尝试运行,但是因为go mod 下载包的原因而不不能通过编译,
我用的是go 12.4
GO111MODULE=on
GOPROXY=https://athens.azurefd.net
tagexpr 使用了包 github.com/henrylee2cn/goutil/tpack
这个包用 go mod tidy 下载不下来
func TestSlice(t *testing.T) {
assert := assert.New(t)
type Request struct {
IDs []int `vd:"$ == nil || len($) > 0"`
}
req := &Request{}
assert.Nil(req.IDs)
err := validator.Validate(req)
assert.Nil(err) // this assert would fail
}
I have one question and one improvement suggestion, the scenario is as below:
AssetListRequestV1
is the request struct
AssetPlatform
field, the validate rule is: all the elements in it should be "1" or "2" or "3" how do I write the validation tag? // asset list request
AssetListRequestV1 struct {
AssetPlatform []string
AmountLow float64 `vd:"$>=0"`
AmountHigh float64 `vd:"$>=(AmountLow)$"`
AprLow int `vd:"$>=0 && $<=100"`
AprHigh int `vd:"$>=0 && $<=100 && $>=(AprLow)$"`
RatingLow int `vd:"$>=0 && $<=100"`
RatingHigh int `vd:"$>=0 && $<=100 && $>=(RatingLow)$"`
TradeCreditLow int `vd:"$>=0 && $<=100"`
TradeCreditHigh int `vd:"$>=0 && $<=100 && $>=(TradeCreditLow)$"`
OrderCond []OrderCond
Page int `vd:"$>=1"`
RowCount int `vd:"$>=1"`
}
OrderCond struct {
OrderField string `vd:"$=='asset_platform' || $=='asset_identifier' || $=='update_ts' || $=='debtor' || $=='amount' || $=='apr' || $=='due_dt' || $=='rating' || $=='trade_credit'"`
Order string `vd:"$=='desc' || $=='asc'"`
}
OrderCond
is the sub field of AssetListRequestV1, although I have write validation tag in its own struct, but it will not check if I wirte code like thisvar vd = validator.New("vd")
func ListAssetDetail(c *gin.Context) {
// get req
req := new(AssetListRequestV1)
if err := c.Bind(req); err != nil {
ResponseErr(c, ErrInvalidParam, err.Error())
return
}
// input check
if err := vd.Validate(req); err != nil {
ResponseErr(c, ErrRequestValidationNotPass, err.Error())
return
}
.........
}
actually, I need to range the nested struct by myself
var vd = validator.New("vd")
func ListAssetDetail(c *gin.Context) {
// get req
req := new(AssetListRequestV1)
if err := c.Bind(req); err != nil {
ResponseErr(c, ErrInvalidParam, err.Error())
return
}
// input check
if err := vd.Validate(req); err != nil {
ResponseErr(c, ErrRequestValidationNotPass, err.Error())
return
}
for _, cond := range req.OrderCond {
if err := vd.Validate(&cond); err != nil {
ResponseErr(c, ErrRequestValidationNotPass, err.Error())
return
}
}
.......
}
I think it can be improved
@henrylee2cn
比如说 gin支持设置默认值:
UserName string form:"user_name,default=wyz"
这个对binding很有用
当入参填写了 default tag 并且没有 json tag 的时候,无论 body 中填什么值,都会被 default 值覆盖。
body 入参的定义如下
type GetItemRequest struct {
Content string thrift:"Content,1,required" default:"hello"
}
此时,curl 请求
curl 'http://localhost:6791?Action=GetItem'
--header 'Content-Type: application/json'
--data-raw '{
"Content": "123",
}'
后,Content 值依然为 hello
猜测原因:
Line 169 in 873eca5
Line 213 in 873eca5
当使用msg自定义错误信息时
validator的Error的行为是,如果有自定义错误信息就只返回自定义信息
// Error implements error interface.
func (e *Error) Error() string {
if e.Msg != "" {
return e.Msg
}
return "invalid parameter: " + e.FailPath
}
而binding的Error却会有其他附加信息
func (e *Error) Error() string {
if e.Msg != "" {
return e.ErrType + ": expr_path=" + e.FailField + ", cause=" + e.Msg
}
return e.ErrType + ": expr_path=" + e.FailField + ", cause=invalid"
}
建议,
placeholders := map[string]string {"{{ ErrType }}":e.ErrType, "{{ FailField }}":e..FailField }
msg := e.Msg
for k, v := range placeholders{
msg = strings.Replace(msg, k, v)
}
return msg
为了减少传输量,对请求body进行了gzip压缩,通过Content-Encoding进行标识。go-tagexpr是否准备支持?
如果需要提交实现PR,是引入一个新的bodycode,还是在binding内对Content-Encoding进行处理?
caused by used package github.com/henrylee2cn/ameda
15:36:40 # github.com/henrylee2cn/ameda
15:36:40 vendor/github.com/henrylee2cn/ameda/int.go:13:2: constant 9223372036854775807 overflows int
15:36:40 vendor/github.com/henrylee2cn/ameda/uint.go:13:2: constant 18446744073709551615 overflows uint
在实际开发中,针对某些字段只能取特定的值,并且这些值不连续。枚举就很有必要了
hi, tagexpr is powerful but the examples use so many variables like a
b
c
d
, which is not meaningful.
I suggest using some true scenario of any kind, say: validate an amount, email, phoneno, etc, which is more interesting and demonstrate the idea.
type TStruct struct {
TOk string `vd:"gt($,'0') && gt($, '1')" json:"t_ok"`
//TFail string `vd:"gt($,'0')" json:"t_fail"`
}
func TInit() {
validator.RegFunc("gt", func(args ...interface{}) error {
return errors.New("force error")
})
}
func Test_tagexpr(t *testing.T) {
TInit()
fmt.Printf("%+v\n", validator.Validate(&TStruct{TOk: "1" /*TFail: "1"*/}))
}
如上代码
自定义了“gt”函数,TFail字段单独使用gt函数,validate会报错,符合预期;
但是TOk字段,对两个gt函数执行了逻辑运算,validate会通过,不符合预期。
github.com/andeya/goutil/
github.com/henrylee2cn/goutil/
你改个名这个玩意就各种报错
When trying to install the newest version, I got the error below:
go get: github.com/henrylee2cn/[email protected] updating to
github.com/henrylee2cn/[email protected]: parsing go.mod:
module declares its path as: github.com/andeya/goutil
but was required as: github.com/henrylee2cn/goutil
IsVisible bool `json:"isVisible" vd:"$!nil"`
I get the error "validating".
现在只能指定验证某个索引的元素,针对那种需要验证切片所有元素的情况下没发操作
proto的oneof特性生成的结构是接口形式的貌似不支持啊!
../github.com/bytedance/go-tagexpr/tagexpr.go:622:14: v.MapRange undefined (type reflect.Value has no field or method MapRange)
想对一个结构体检查,只要有一个参数不在预期就返回false
伪代码如下
type T struct {
A int `tagexpr:"$<0||$>=100"`
B string `tagexpr:"len($)>1 && regexp('^\\w*$')"`
C bool `tagexpr:"{expr1:(f.g)$>0 && $}{expr2:'C must be true when T.f.g>0'}"`
d []string `tagexpr:"{@:len($)>0 && $[0]=='D'} {msg:sprintf('invalid d: %v',$)}"`
e map[string]int `tagexpr:"len($)==$['len']"`
e2 map[string]*int `tagexpr:"len($)==$['len']"`
f struct {
g int `tagexpr:"$"`
}
}
vm := tagexpr.New("tagexpr")
err := vm.WarmUp(new(T))
if err != nil {
panic(err)
}
t := &T{
A: 107,
B: "abc",
C: true,
d: []string{"x", "y"},
e: map[string]int{"len": 1},
e2: map[string]*int{"len": new(int)},
f: struct {
g int `tagexpr:"$"`
}{1},
}
tagExpr, err := vm.Run(t)
if err != nil {
panic(err)
}
// 看我
b := tagExpr.CheckAll(t)
if !b {
fmt.Printf("invalid argument\n")
}
type struct A {b *B}
type struct B {a *A}
will cause stack over flow, think of cycle cite check?
type Address struct{
City string `form:"city,required" json:"city,required" query:"city,required"`
}
type User struct{
Name string `form:"name" json:"name,required" query:"name"`
Address *Address `form:"address" json:"address,omitempty" query:"address"`
}
// request body 为json {"name":"Jack", "address":null}
var user model.User
err = binding.BindAndValidate(c, &user)
预期user能正常的绑定,user.Address == nil,实际上返回错误提示 binding: expr_path=address.city, cause=missing required parameter
For basic types like int8
, we can define some string enumerations for those types.
Can RegTypeUnmarshal
allow those types, restrict Unmarshal
for real basic types, not user-defined types with underlying basic types?
Hi
Is it possible to support custom JSON tag for binding data? As sometime for the same data, we will want to support snake case and camel case for the same field.
Example:
type Person struct {
DateOfBirth string `json:"dateOfBirth" sjson:"date_of_birth"`
}
go-tagexpr/binding/gjson/gjson.go
Line 47 in a44f579
Hi there!
I would be great, of binder would support to bind a query parameter to an struct.
Example:
type query struct {
a struct {
a string
b string
c string
d string
} `query:"a"`
}
and a[a]=b
would map to query.a.a
. This would be helpful, if my own application includes structs from 3rd party vendors, where I cant attach additional query
metadata.
如在
`query:"profiling_id" vd:"@:len($)==32; msg:'profiling_id's length is invalid'"`
这样的句子里,msg 里包含的单引号会使整个vd失效而无法产生正确结果; 如果将msg 里的单引号转义 ',会产生一个更奇葩的效果就是服务第一次的验证会成功,而之后的验证都会失效。
以及同学有没有内部的联系方式方便讨论呀?
func (b *Binding) bindStruct(structPointer interface{}, structValue reflect.Value, req Request, pathParams PathParams) (hasVd bool, err error) {
recv, err := b.getOrPrepareReceiver(structValue)
if err != nil {
return
}
expr, err := b.vd.VM().Run(structValue)
if err != nil {
return
}
bodyCodec, bodyBytes, err := recv.getBodyInfo(req)
if len(bodyBytes) > 0 {
err = b.prebindBody(structPointer, structValue, bodyCodec, bodyBytes)
}
if err != nil {
return
}
bodyString := ameda.UnsafeBytesToString(bodyBytes)
postForm, err := req.GetPostForm()
if err != nil {
return
}
var fileHeaders map[string][]*multipart.FileHeader
if _req, ok := req.(requestWithFileHeader); ok {
fileHeaders, err = _req.GetFileHeaders()
if err != nil {
return
}
}
queryValues := recv.getQuery(req)
cookies := recv.getCookies(req)
headers := req.GetHeader()
for _, param := range recv.params {
for i, info := range param.tagInfos {
var found bool
switch info.paramIn {
case raw_body:
err = param.bindRawBody(info, expr, bodyBytes)
found = err == nil
case path:
found, err = param.bindPath(info, expr, pathParams)
case query:
found, err = param.bindQuery(info, expr, queryValues)
case cookie:
found, err = param.bindCookie(info, expr, cookies)
case header:
found, err = param.bindHeader(info, expr, headers)
case form, json, protobuf:
if info.paramIn == in(bodyCodec) {
found, err = param.bindOrRequireBody(info, expr, bodyCodec, bodyString, postForm, fileHeaders,
recv.hasDefaultVal)
} else if info.required {
found = false
err = info.requiredError
}
case default_val:
found, err = param.bindDefaultVal(expr, param.defaultVal)
}
if found && err == nil {
break
}
if (found || i == len(param.tagInfos)-1) && err != nil {
return recv.hasVd, err
}
}
}
return recv.hasVd, nil
}
你好!
var (
jsonUnmarshalFunc func(data []byte, v interface{}) error
)
// ResetJSONUnmarshaler reset the JSON Unmarshal function.
// NOTE: verifyingRequired is true if the required tag is supported.
func ResetJSONUnmarshaler(fn JSONUnmarshaler) {
jsonUnmarshalFunc = fn
}
现在 jsonUnmarshalFunc 是全局变量,ResetJSONUnmarshaler 是全局影响的函数。
可否将 jsonUnmarshalFunc 绑定到 Binding 结构体上,然后 ResetJSONUnmarshaler 函数,去修改 defaultBinding 实例?
这样的话,可以控制只影响一个实例的 unmarshal 效果。
如果可以的话,我可以来提供这个修改。谢谢。
比如说
startDate time.Time `json:"start_date,layout=2016-01-02"`
支持time.Time类型的field的binding,并且支持用户自定义layout
it is so powerful.👍
it might reduce code readability when tag is so long.
how to express switch - case
case
比如
type Key string
sortKey Key `query:"sort_key"`
这种binding现在是不支持的
报错: panic: reflect.Set: value of type string is not assignable to type binding_test.metric
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.