ezbuy/redis-orm

write db yaml once, generate go orm code everywhere.

redis-orm

redis-orm fly orm up

Important NOTE

redis-orm is not maintained anymore , try to use the ezorm's mysqlr driver instead.

features

quick start

generate command

go get github.com/ezbuy/redis-orm

redis-orm code -i example/yaml -o example/model

read access usage

import "github.com/ezbuy/redis-orm/example/model"

// mysql
model.MySQLSetup(cf)


db := model.MySQL()
// query (ids []string) by unique & index & range definitions
model.UserDBMgr(db).FindOne(unique)
model.UserDBMgr(db).Find(index)
model.UserDBMgr(db).Range(scope)
model.UserDBMgr(db).RangeRevert(scope)

// search usage
blogs, err := model.BlogDBMgr(db).Search("INNER JOIN users ON blog.user_id = users.user_id WHERE users.mail_box=?", "", "", "[email protected]")

// fetch object 
model.UserDBMgr(db).Fetch(pk PrimaryKey) (*User, error)
model.UserDBMgr(db).FetchByPrimaryKey(id int32) (*User, error)
model.UserDBMgr(db).FetchByPrimaryKeys(ids []int32) ([]*User, error)

// redis
model.RedisSetup(cf)

redis := model.Redis()
// query (ids []string) by unique & index & range definitions
model.UserRedisMgr(redis).FindOne(unique)
model.UserRedisMgr(redis).Find(index)
model.UserRedisMgr(redis).Range(scope)
model.UserRedisMgr(redis).RangeRevert(scope)

// fetch object 
model.UserRedisMgr(redis).Fetch(pk PrimaryKey) (*User, error)
model.UserRedisMgr(redis).FetchByPrimaryKeys(pks []PrimaryKey) ([]*User, error)

write access usage

import "github.com/ezbuy/redis-orm/example/model"

// mysql
model.MySQLSetup(cf)

db := model.MySQL()
tx, err := db.BeginTx()
defer tx.Close()

model.UserDBMgr(tx).Save(obj)
model.UserDBMgr(tx).Create(obj)
model.UserDBMgr(tx).Update(obj)
model.UserDBMgr(tx).Delete(obj)

model.UserDBMgr(tx).FindOne(unique)
model.UserDBMgr(tx).Find(index)
model.UserDBMgr(tx).Range(scope)
model.UserDBMgr(tx).RangeRevert(scope)

model.UserDBMgr(tx).Fetch(id string) (*User, error)
model.UserDBMgr(tx).FetchByPrimaryKey(id int32) (*User, error)
model.UserDBMgr(tx).FetchByPrimaryKeys(ids []int32) ([]*User, error)

// redis
model.RedisSetup(cf)

redis := model.Redis()
model.UserRedisMgr(redis).Save(obj)
model.UserRedisMgr(redis).Create(obj)
model.UserRedisMgr(redis).Update(obj)
model.UserRedisMgr(redis).Delete(obj)

sync data

import "github.com/ezbuy/redis-orm/example/model"

model.MySQLSetup(cf)
model.RedisSetup(cf)

db := model.MySQL()
redis := model.Redis()

model.UserRedisMgr(redis).Load(model.UserDBMgr(db))

bench redis vs mysql

enviroment:

mysql-server, redis-server, test client all in the same machine (mac air)

redis-orm.redis.bench

Ran 1000 samples:
unique.runtime:
  Fastest Time: 0.000s
  Slowest Time: 0.001s
  Average Time: 0.000s ± 0.000s
index.runtime:
  Fastest Time: 0.000s
  Slowest Time: 0.000s
  Average Time: 0.000s ± 0.000s
range.runtime:
  Fastest Time: 0.000s
  Slowest Time: 0.000s
  Average Time: 0.000s ± 0.000s
range.revert.runtime:
  Fastest Time: 0.000s
  Slowest Time: 0.000s
  Average Time: 0.000s ± 0.000s
fetch.runtime:
  Fastest Time: 0.002s
  Slowest Time: 0.004s
  Average Time: 0.002s ± 0.000s

redis-orm.mysql.bench

Ran 1000 samples:
unique.runtime:
  Fastest Time: 0.002s
  Slowest Time: 0.106s
  Average Time: 0.003s ± 0.005s
index.runtime:
  Fastest Time: 0.002s
  Slowest Time: 0.106s
  Average Time: 0.003s ± 0.005s
range.runtime:
  Fastest Time: 0.002s
  Slowest Time: 0.105s
  Average Time: 0.002s ± 0.005s
range.revert.runtime:
  Fastest Time: 0.002s
  Slowest Time: 0.105s
  Average Time: 0.002s ± 0.006s
fetch.runtime:
  Fastest Time: 0.004s
  Slowest Time: 0.150s
  Average Time: 0.006s ± 0.009s

redis-orm's Issues

proposal: pb plugin

Background

At ezbuy, we use grpc protobuf as our protocol between all of ours ends. However, as for our backend(Go) team, we always have two main data structure, one is the DB scheme struct(generated by redis-orm), the other one is the pb scheme(generated by protoc), so , we always write some conversion wrappers , like:

modelA := new(model.A)
// do sth to fill the modelA
pbA := &pb.A{
  AA: modelA.A,
  AB: modelA.B,
// etc...
}

Once the two struct becomes huge, our wasted time will also be boomed.

Proposal

What about add the pb plugin into the redis-orm ?

Blog:
  dbs: [mysql]
  dbname: ezorm
  dbtable: blogs
  fields:
    - Id: int32
    - UserId: int32
       pb: [UserID]
    - title: string
      es_do_index: true
      pb: [Title]
    - Content: string
      es_analyzer: standard
    - Status: int32
      flags: [index]
    - Readed: int32
    - CreatedAt: timestamp
      es_do_index: true
    - UpdatedAt: timestamp
  primary: [Id, UserId]

And then, it will generate the extra pb conversion code like:

package xxx

// ConvertModelToPB converts model to defined pb columns.
func ConvertModelToPB(md *model.Blog)*pb.Ezorm{
  return &pb.Ezorm{
     UserID: md.UserId,
     Title: md.Titel, 
 }
}

// ConvertPBToModel converts defined pb columns to model. 
func ConvertPBToModel(p *pb.Ezorm)*model.Blog{
 return &model.Blog{
   UserId: p.UserID,
   Title: p.Title,
 }
}

Benefits

• Decrease our wasted time on writing duplicated code.
• Decrease bugs caused by matching columns incorrectly.

Risk

• it will enlarge our generate codebase, and have no solutions to prevent abuse.

Break compatibility ？

No.

redis: ranges generate ZSet spam into redis table

Issue Description

With the yaml definition like this:

Blog:
  dbs: [redis]
  dbname: blog
  dbtable: user_blog
  fields:
    - ID: int64
      flags: [index]
    - Content: string
    - CreateDate: int64
    - UpdateDate: int64
primary: [ID]

In object.go, it will pass the o.primary.IsRange(), and create o.ranges:

if o.primary.IsRange() {
	index := NewIndex(o)
	index.FieldNames = o.primary.FieldNames
	o.ranges = append(o.ranges, index)
}
---
func (pk *PrimaryKey) IsRange() bool {
	c := len(pk.Fields)
	if c > 0 {
		return pk.Fields[c-1].IsNumber()
	}
	return false
}

And then, once we invoke our orm function addToPipeline, it will save the range info into redis table:

	//! ranges
	rg_key_0 := []string{
		"Id",
	}
	rg_pip_0 := IdOfUserRNGRelationRedisMgr().BeginPipeline(pipe.Pipeline)
	rg_rel_0 := IdOfUserRNGRelationRedisMgr().NewIdOfUserRNGRelation(strings.Join(rg_key_0, ":"))
	score_rg_0, err := orm.ToFloat64(obj.Id)
	if err != nil {
		return err
	}
	rg_rel_0.Score = score_rg_0
	rg_rel_0.Value = key
	if err := rg_pip_0.ZSetAdd(rg_rel_0); err != nil {
		return err
	}
	if expire > 0 {
		pipe.Expire(keyOfObject(obj, key), expire)
	}

But, we do not need this kind of range, we only just need the primary mechanism .

What 's more worse is that we store the 50%+ redis ranges(spam) into our redis , and they all have no TTL !!!

Workarounds

How about add the norange flag into our redis column definition ?

Columns with norange flag will quit the range code generation, and then we can change our primary.key.go into this:

func (pk *PrimaryKey) IsRange() bool {
      var fs []*Fields
      for _,f := range pk.Fields{
          if  !f.IsRange(){
            continue
         }
          fs = append(fs,f)
       }
   	c := len(fs)
	if c > 0 {
		return pk.Fields[c-1].IsNumber()
	}
  return false
}

MySQL: orm should support context

background

In our daily development, we use the SQL operations together with some rpc functions . And, we pass the context between the rpc invocation , however, redis-orm do not accept this ctx within its SQL execution.

proposal

	result, err := tx.tx.Exec(sql, args...)
	if err != nil {
		tx.err = err
	}

we should provide a new method to pass the outside ctx into the Exec , perhaps it will be like :

	result, err := tx.tx.ExecContext(ctx,sql, args...)
	if err != nil {
		tx.err = err
	}

covered situation

• grpc: The invoker inspector client canceled(OR Timeout) the grpc request,and set the context,but the server side will continue the SQL operations .And once the invoker retries , it will receive some database error(i.e. the record lock error OR the dirty reads)

tpl: support sql.ErrNoRows

proposal

We should modify the unspecific error fmt.Errorf("OperXXX has no rows") into the certain error type sql.ErrNoRows in the most FetchXXX funcs.

And wraps this error with fmt.Errorf("DBTable: %w", sql.ErrNoRows)

advantages

With this patch , we can simply use errors.Is(err,sql.ErrNoRows) to check if it is a no rows error in the upstream codes.

parser: yaml field `sqlcolumn` not set db field

Reproduce

foo. yaml:

Foo:
ID: int64
sqlcolumn: ID

gen.foo.go:

type Foo struct{
	ID  int64 `db:"id"`
}

parser: convert yaml defined Time to Go 1.13 new database/sql type NullTime

See Go 1.13 release note: https://golang.org/doc/go1.13#database/sql