douyu/jupiter

Jupiter: Governance-oriented Microservice Framework.

JUPITER: Governance-oriented Microservice Framework

Introduction

JUPITER is a governance-oriented microservice framework, which is being used for years at Douyu.

Online Demo

Jupiter Console (Juno)

Username: admin
Password: admin

Documentation

See the 中文文档 for the Chinese documentation.

Requirements

• Go version >= 1.19
• Docker

Quick Start

1. Install jupiter toolkit
2. Create example project from jupiter-layout
3. Download go mod dependencies
4. Run the example project with jupiter toolkit
5. Just code yourself :-)

go install github.com/douyu/jupiter/cmd/jupiter@latest
jupiter new example-go
cd example-go
go mod tidy
docker-compose -f test/docker-compose.yml up -d
jupiter run -c cmd/exampleserver/.jupiter.toml

Learn More:

• Juno: Microservice Governance System for jupiter
• Jupiter Layout: Project Template for jupiter
• Examples

Bugs and Feedback

For bug report, questions and discussions please submit an issue.

Contributing

Contributions are always welcomed! Please see CONTRIBUTING for detailed guidelines.

You can start with the issues labeled with good first issue.

Contact

• Wechat Group

jupiter's Issues

support redis master-slave and fragmented redis switching functions

使用场景：

在本地使用的单个Redis：
redisStub := redis.StdRedisConfig("myredis").Build()

在正式环境使用的是Redis集群：
redisStub := redis.StdRedisClusterConfig("myredis").Build()

这样就产生一个问题，在代码里使用Redis的地方都需要判断是本地环境还是线上正式环境。
在框架中是否可以进一步封装，对于使用者不用管是什么环境，只需要传一个参数来确定是否为Redis集群？或者有更好的方案？

[RFC-20200602-01] 开发流程提议

简介

为了规范化开发,推动社区加入贡献,特编写本提案. 旨在阐述, 如何规范化开发流程

流程

1. 编写RFC 文档, 提交issue: 如果你想改变什么功能, 新增什么功能, 首先需要有一个想法
2. 开发参考实现, 提交pull request
3. 等待开发者审核,确认
4. 如果接受了您的提案, 开发者会跟你讨论 pull request的详细细节
5. 如果所有的代码, 有任何问题, 会被要求修正, 或者被拒绝
6. 没有任何意外情况, pull request 会被合并
7. RFC 被标记为解决,并关闭

备注

1. 开发团队拥有最终解释权
2. 您的RFC可能不被接受,请理解这一点
3. 如果没有一个高品质的参考实现,仅有RFC, 是不足够的
4. 参考实现代码,必须有对应的测试用例,没有错误
5. 贡献者应该及时关注更新进展

supports facebookincubator/ent as orm

Issue Description

Type: feature request

Describe what feature you want

ent 是 facebook一个很好的orm lib， 目前也非常成熟了。

考虑是否支持。

Additional context

Add any other context or screenshots about the feature request here.

求一个apigateway的例子

Issue Description

希望 增加 apigateway 的事例
Type: feature request

Describe what feature you want

Additional context

Add any other context or screenshots about the feature request here.

[BUG]xgin trace ctx

Issue Description

Type: bug report

Describe what happened

jupiter/pkg/server/xgin/middleware.go

Line 186 in a1058be

c.Request.WithContext(ctx)

这行代码没有将ctx添加到Request中，可以修改为：c.Request = c.Request.WithContext(ctx)

Describe what you expected to happen

How to reproduce it (as minimally and precisely as possible)

Tell us your environment

Anything else we need to know?

supports dynamic port

Issue Description

Type: feature request

Describe what feature you want

支持动态端口，配置文件port=0时，由系统自动生成端口并能注册正确端口信息到etcd

Additional context

Add any other context or screenshots about the feature request here.

[BUG] go get 会报错 invalid argument newArgs

go get -v github.com/douyu/jupiter/tools/jupiter

douyu/jupiter/tools/jupiter/new/tools.go:35:8: invalid argument newArgs (type cli.Args) for len
douyu/jupiter/tools/jupiter/new/tools.go:39:17: invalid operation: newArgs[0] (type cli.Args does not support indexing)
douyu/jupiter/tools/jupiter/main.go:31:15: cannot use []cli.Command literal (type []cli.Command) as type []*cli.Command in assignment
会报错，望尽快修复

[BUG]redis客户端HIncrBy未返回值，总共两个地方啊，你们只修了一个

Issue Description

Type: bug report

Describe what happened

总共两个地方啊啊啊啊

  redis_cmds.go

和

cluster_cmds.go

你们修了 cluster_cmds.go
没修 redis_cmds.go

Describe what you expected to happen

How to reproduce it (as minimally and precisely as possible)

Tell us your environment

Anything else we need to know?

[BUG]基于ETCD的GRPC服务动态端口的问题

基于ETCD的GRPC服务动态端口的问题：

如果GRPC服务基于ETCD，配置如下：
[jupiter.server.grpc]
port = 0

如果先启动客户端，再启动服务端，现象同#64【基于ETCD的GRPC服务】一样的错误。

再正式环境中，经常会出现只更新某个服务的现象。

examples about grpc load balancing

Issue Description

Type: feature request

Describe what feature you want

more examples and demo about grpc load balancing with group and weight

Additional context

Add any other context or screenshots about the feature request here.

Windows configuration path resolution bug

Issue Description

Type: bug report

Describe what happened

When using Jupiter, the following error occurs when Windows launches the specified configuration file:
panic:
msg: new engine
errVerbose: runtime error: slice bounds out of range [:-1]
github.com/douyu/juno-agent/vendor/github.com/douyu/jupiter.(*Application).loadConfig-fm

Describe what you expected to happen

Can support Windows system specified configuration file startup

How to reproduce it (as minimally and precisely as possible)

1. Execute the following command at the terminal: juno-agent --config=../../config/config.live.toml

Tell us your environment

os: windows 8.1
go: 1.13.9

Anything else we need to know?

[BUG]基于ETCD的GRPC服务

以下场景是基于ETCD的GRPC服务：

在GRPC服务端启动后，再启动GRPC客户端，一切正常。但是在此时把GRPC服务端重新启动后，GRPC客户端不管等多久，GRPC客户端都连接不上GRPC服务端。

Jupiter scaffold supports grpc and swagger

Issue Description

Type: feature request

Describe what feature you want

Let Jupiter generate from Pb file pb.go File, support optional generation of client and server code, support online swagger

Additional context

Add any other context or screenshots about the feature request here.

support config center?

Have you considered supporting the microservice configuration center？
eg：apollo

optimize return value of redis client

Issue Description

Type: bug report

Describe what happened

redis_cmds.go

HIncrBy 哈希field自增
func (r *RedisStub) HIncrBy(key string, field string, incr int) {
	r.Client.HIncrBy(key, field, int64(incr))
}

Describe what you expected to happen

How to reproduce it (as minimally and precisely as possible)

Tell us your environment

Anything else we need to know?

不支持熔断

希望能支持熔断和服务降级

support apollo configuration center for datasource

Issue Description

Type: feature request

Describe what feature you want

support Apollo configuration center for datasource

Additional context

Add any other context or screenshots about the feature request here.

update security policy component to sentinel-golang 0.3.0

Issue Description

Type: feature request

Describe what feature you want

update ratelimit/circuit-breaker to sentinel-golang 0.3.0

Additional context

Add any other context or screenshots about the feature request here.

[BUG]快速开始文档有错误

Issue Description

Type: bug report

there are few mistakes in quickstart.md

go get github.com/douyu/jupiter/tools/jupiter 报错

[BUG]主动断开( ctrl + c) http server 控制台会报错

Issue Description

主动断开( ctrl + c) http server 控制台会报错

Describe what happened

运行的是 example/http/gin/main.go

go run main.go --config=config.toml

support self-defined logger? like datetime format, timestamp seem not so friendly

Issue Description

Type: feature request

Describe what feature you want

Additional context

Add any other context or screenshots about the feature request here.

工程目录对swagger不友好的问题

目前自动生成的目录工程main.go在demo/cmd下面，这种情况下面我执行swag init的时候无法自动化生成文档，我将main.go移动到demo目录下面即可，使用的框架是gin以及gin-swagger

Improve metrics for those components that are missing metrics collection.

Issue Description

Type: feature request

Describe what feature you want

Improve metrics for those components that are missing metrics collection.

Additional context

Add any other context or screenshots about the feature request here.

[BUG]support the signal of windows system and oxs system

Issue Description

在window环境下，项目编译报错，希望兼容window信号处理；

Type: bug report

Describe what happened

go build .
`

github.com/douyu/jupiter

..\jupiter\signal.go:34:3: undefined: syscall.SIGSTOP
..\jupiter\signal.go:35:3: undefined: syscall.SIGUSR1
..\jupiter\signal.go:36:3: undefined: syscall.SIGUSR2
..\jupiter\signal.go:46:26: undefined: syscall.SIGSTOP
..\jupiter\signal.go:46:43: undefined: syscall.SIGUSR1
..\jupiter\signal.go:50:42: undefined: syscall.SIGUSR2
note: module requires Go 1.14

`

Describe what you expected to happen

提供window信号处理支持

How to reproduce it (as minimally and precisely as possible)

1. 在window环境下使用jupiter框架

Tell us your environment

set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\xubing\AppData\Local\go-build
set GOENV=C:\Users\xubing\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\xubing\go
set GOPRIVATE=
set GOPROXY=https://goproxy.cn,direct
set GOROOT=C:\Go
set GOSUMDB=off
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\xubing\AppData\Local\Temp\go-build430993566=/tmp/go-build -gno-record-gcc-switches

Anything else we need to know?

建议增加脚手架工具，自动生成项目目录

建议增加脚手架工具，自动生成项目目录。并且可以弄一套比较完整的功能点完整的demo

建议增加swagger示例或自动生成文档功能

建议增加swagger示例或自动生成文档功能

[BUG] go get -u github.com/douyu/jupiter/tools/jupiter error

Issue Description

Type: bug report
go get -u github.com/douyu/jupiter/tools/jupiter error

Describe what happened

Describe what you expected to happen

How to reproduce it (as minimally and precisely as possible)

Tell us your environment

Anything else we need to know?

[BUG] 文档更新，并没有使用redigo，你们用的go-redis

Improve tracing with more tags and labels

Issue Description

Type: feature request

Describe what feature you want

ref: https://github.com/opentracing-contrib/opentracing-specification-zh/blob/master/semantic_conventions.md

Additional context

Add any other context or screenshots about the feature request here.

[BUG] 执行 go get -u github.com/douyu/jupiter/tools/jupiter 报错

Issue Description

Type: bug report

Describe what happened

➜ tmp go get -u github.com/douyu/jupiter/tools/jupiter
# github.com/douyu/jupiter/tools/jupiter/new
../../../go/src/github.com/douyu/jupiter/tools/jupiter/new/tools.go:35:8: invalid argument newArgs (type cli.Args) for len
../../../go/src/github.com/douyu/jupiter/tools/jupiter/new/tools.go:39:17: invalid operation: newArgs[0] (type cli.Args does not support indexing)

Describe what you expected to happen

执行完 go get -u github.com/douyu/jupiter/tools/jupiter 后 jupiter 命令可用

How to reproduce it (as minimally and precisely as possible)

Tell us your environment

Anything else we need to know?

[ci] 用actions 替换TravisCI

Issue Description

是否有计划用actions 完全取代 TravisCI
感觉actions 速度上比 TravisCI更好点 ，另外也是github原生的。

Type: feature request

Describe what feature you want

如果可以通过，我将尝试按照 .travis.yml 的功能 补充 .github/workflows/go.yml文件

Additional context

Add any other context or screenshots about the feature request here.

[BUG]中文简介页面排版问题

Issue Description

Type: bug report

Describe what happened

这里应该是无序列表的第一项吧，按照issue#15中的方式没能提pr，只好新建一个issue。

bytes.Buffer for some long string or byte concatenation, it can indeed improve performance, but don’t forget that this will open up a space of const smallBufferSize = 64 size

jupiter/pkg/ecode/status.go

Line 72 in 2548643

var buf bytes.Buffer

bytes.Buffer for some long string or byte concatenation, it can indeed improve performance, but don’t forget that this will open up a space of const smallBufferSize = 64 size, I found that many of your code in golang are written like this, if for comparison A small amount of splicing, please do not open up so much space, this is a waste, it is better to use slice append+strings.Join to complete, please do not write some more poke code, please

微信群二维码扫描过期了

Issue Description

Type: feature request

Describe what feature you want

Additional context

Add any other context or screenshots about the feature request here.

support Gin framework

Issue Description

Type: feature request

Describe what feature you want

Additional context

Add any other context or screenshots about the feature request here.

建议新增对goframe框架的内置支持

在“1.3.3 多服务端框架”建议增加对goframe框架的内置支持

yaml拼写错误？

是不是拼错了？

Import CloudEvent to spec event message between of framework, agent and governace platform

Issue Description

Type: feature request

Describe what feature you want

Additional context

Add any other context or screenshots about the feature request here.

[feat] support more grpc load balance algorithms.

Issue Description

Type: feature request

Describe what feature you want

• p2c: least loaded
• p2c: peak ewma
• aperture
• ...

Additional context

Add any other context or screenshots about the feature request here.

unified application metadata management

Issue Description

Type: feature request

Describe what feature you want

内部环境的部分应用元数据不通用，需要重新设计jupiter的元数据。

以便支持指标采集、负载均衡、链路染色等功能。

Additional context

Add any other context or screenshots about the feature request here.

hookSignals func,these two places can be combined into a case,Sleep paused for 3s, in fact, you can use the Context cancel mechanism here.

jupiter/signal_linux.go

Line 46 in 2548643

case syscall.SIGQUIT, syscall.SIGSTOP, syscall.SIGUSR1:

jupiter/signal_linux.go

Line 50 in 2548643

case syscall.SIGINT, syscall.SIGKILL, syscall.SIGUSR2, syscall.SIGTERM:

These two places can be combined into one case. In this for, semaphore monitoring is performed. Although an independent goroutine+for is continuously opened to ask whether the semaphore is monitored, it uses a very time method. Sleep paused for 3s, in fact, you can use the Context cancel mechanism here. There is another bad point here. Since the smooth exit is done, how do you know that it is paused for 3s? I think that the smooth exit should be done through the context ctx mechanism or through the channel, or you can just wait for the context ctx to timeout and exit. Are better than direct time.Sleep 3s

关于调整注册时机与监听时机的问题

Issue Description

Type: feature request

相关pr
#58

1，调整了servers 注册、注销时机（将注销放在了stop中，将注册测defer挪到了eg.go外的线程中）
2，调整了注册服务的关闭时机（放在了最后）
3，统一调整了server下的 listen时机 （统一放在了serve方法中，newServer不在承担打开监听的事）

Describe what feature you want

现在有过程大致是这样的：

1，server.newServer 主要读取配置初始化等事情
2，注册服务 （未启动服务之前就注册服务）
3，server.Serve 主要做启动服务的事情
4, 注销服务（server线程停止后注销）
5, server.Stop 主要做停止的事情

启动一个网络服务一般会有分两步 listen 和serve，若将这两部全部放在 Serve环节
将有个问题，就是注册服务，无法正确拿到启动信息，如：实际启动端口 （动态端口时）
另外，注册的时机，比启动的时机稍微提前。

调整后：

1，server.newServer
2，server.Serve
3，注册服务 （待Serve线程启动后，等待1毫秒后注册）
4, server.Stop
5，注销服务 （在触发server.Stop 后进行注销服务）

即在Serve 启动服务后，进行服务注册。在服务stop后进行服务注销。

这样调整后，注册服务时，获取的server.info 为Serve后的状态，即可以取到动态端口号等信息。

因调整涉及注册时机，所以影响较大，请评估风险，以及可能遇到的问题。

Additional context

过程中参考了gomicro 的注册机制，是在listen之后进行的

[BUG]HTTP port occupied

Issue Description

Type: bug report

Describe what happened

When the HTTP port is occupied, the program hangs and there is no log prompt

Describe what you expected to happen

When the HTTP port is occupied, give a prompt and end the program

How to reproduce it (as minimally and precisely as possible)

Using xecho module to open multiple HTTP services

Tell us your environment

Anything else we need to know?

[BUG] xtime包下time.go的 EndOfDay EndOfHour 方法

Issue Description

Type: bug report

Describe what happened

Describe what you expected to happen

func (t *Time) EndOfDay() *Time { y, m, d := t.Date() return &Time{time.Date(y, m, d, 23, 23, 59, int(time.Second-time.Nanosecond), t.Location())} }

第二个23应该为59吧。

func (t *Time) EndOfHour() *Time { y, m, d := t.Date() return &Time{time.Date(y, m, d, t.Hour(), 23, 59, int(time.Second-time.Nanosecond), t.Location())} }

第一个23应该为59吧。

How to reproduce it (as minimally and precisely as possible)

Tell us your environment

Anything else we need to know?

[BUG] unit-test case failed

Issue Description

Type: bug report

Describe what happened

xtime/xgo unit case failed every now and then, which causes some PR CI fail.

unit testing case needs improvement to avoid data race

Describe what you expected to happen

How to reproduce it (as minimally and precisely as possible)

Tell us your environment

Anything else we need to know?

[BUG] 基于jupiter编写的程序以后台方式运行自动退出

Issue Description

Type: bug report

Describe what happened

通过shell远程连接服务器，然后使用nohup启动服务，shell断开进程就退出，只要基于jupiter编写的程序都会，如juno、juno-agent、example\helloworld

Describe what you expected to happen

How to reproduce it (as minimally and precisely as possible)

Tell us your environment

Ubuntu 18.04

Anything else we need to know?

退出原因是shell断开会触发hangup信号，而框架对hangup信号处理方式为退出服务

建议增加权限管理casbin的示例或支持

建议增加权限管理casbin的示例或支持

configuration driver govern server

Issue Description

configuration driver govern server
The govern server can be configured as standard through the framework，let's be better and easier to use govern server

Describe what feature you want

Additional context

Add any other context or screenshots about the feature request here.

issue bug template add env info, refer to golang/go repo

Tell us your environment

What version of Go are you using (go version)?

$ go version

What operating system and processor architecture are you using (go env)?

go env Output

$ go env