toktik-team/toktik

ChatGPT integrated short video microservice application built with Kitex and Hertz. 第五届字节跳动青训营后端大作业“码如磐石奖（一等奖）”获奖项目

Toktik

	ChatGPT integrated short video microservice application built with Kitex and Hertz , made by Toktik-Team in The 5th Bytedance Youth Training Camp.

English | 简体中文

The 5th Bytedance Youth Training Camp Report Documentation: https://qrggpk56b9.feishu.cn/docx/Djwpdm8oZo3JyRx0JwkcZhRDnVd

Recommend

Now Recommend for GuGoTik !

TokTik is no longer maintained and has no planned maintenance plans, please go to GuGoTik. GuGoTik respects and follows TokTik's workline, improving and optimizing almost every components from it and adding more features. For best expermental experience, please use GuGoTik instead of TokTik.

Give us your stars if you like our project.

Awards

Earned the Best Project award in the 5th Bytedance Youth Training Camp.

Demo

Download Dousheng app from here, and use this URL to test with the demo.

https://toktik.xctra.cn/

Project Structure

• constant
  • biz - Business logic related constants
  • config
    • env.go - Environment variable configs
    • service.go - Service names and ports
• idl
  • auth.proto - RPC definition of auth service
  • comment.proto - RPC definition of comment service
  • favorite.proto - RPC definition of favorite service
  • feed.proto - RPC definition of feed service
  • publish.proto - RPC definition of publish service
  • relation.proto - RPC definition of relation service
  • user.proto - RPC definition of user service
  • wechat.proto - RPC definition of chat service
• kitex_gen - Generated code by Kitex
• logging - Logging middleware
• manifests-dev - Kubernetes manifests
• repo - Database schemas and generated code by Gorm Gen
• service
  • auth - Auth service impl
  • comment - Comment service impl
  • favorite - Favorite service impl
  • feed - Feed service impl
  • publish - Publish service impl
  • relation - Relation service impl
  • user - User service impl
  • web - Web api gateway
    • mw - Hertz middlewares
  • wechat - Chat service impl
• storage - Storage middleware, s3, local storage & volcengine ImageX supported
• test
  • e2e - End-to-end test
  • mock - Mock data for unit test

Prerequisite

This project does not support Windows, as Kitex does not support either.

• Linux / MacOS
• Go
• FFmpeg
• PostgreSQL
• Redis
• OpenTelemetry Collector

For observability infrastructures, it's recommended to use:

• Jaeger All in one
• Victoria Metrics
• Grafana

Build

Run ./build-all.sh in your Linux environment to compile all services.

Configurations

Check out constant/config/env.go

Run

• Run start.sh --service <service_name> to start a service.
• service_name could be any of the sub directory of ./service.

Test

Unit Test

Run ./unit-test.sh

End-to-End Test

Run go test toktik/test/e2e -tags="e2e"

How to Contribute

1. Please following the HowToAddServices to create your own service.
2. Create a new branch and make your changes.
3. Create a pull request to the main branch.
4. Wait for review and merge.

Contributors

• Nico
• Eric_Lian
• Dark Litss
• HikariLan
• YunShu

License

Toktik is licensed under the MIT License.

toktik's Issues

Error handling with Kitex Biz Exception.

By default, when an RPC function returns an error, the details of the returned error are lost, and only a string value from err.Error() is kept with the client.

For my perspective, the biz error passing should be improved using Kitex Biz Exception in the future, but for now we should try to handle as many cases as possible in the RPC server function.

I will integrate it in the next sprint, please feel free to start if you are intrested.

Message service session persistency & search with OpenSearch

Feed service rule-based recommendation.

Unit test improvement.

We have several aspects of deficiencies related to unit tests:

• CI Integration
• Coverage calculation & presentation
• Mocking
  • DB
  • Storage
  • RPC Service
• Edge cases
• Fuzzy test

Refactor: add QueryVideosRequest

message QueryVideosRequest {
  uint32 actor_id = 1;
  repeated uint32 video_ids = 2;
}

message QueryVideosResponse {
  uint32 status_code = 1;
  optional string status_msg = 2;
  repeated Video video = 3;
}

service FeedService {
  rpc ListVideos(ListFeedRequest) returns (ListFeedResponse);
  rpc QueryVideos(QueryVideosRequest) returns (QueryVideosResponse);
}

Replace SQL Query to RPC on feed/user/favorite services etc.

Optimize error message send to frontend

• RelationAction
• Auth
• FollowAction

Further AIGC integration

Sync with new API change

In douyin_message_chat_request:

• pre_msg_time

In Message message:

• create_time

In User:

• avatar
  • database
  • service impl (gravatar/cravatar)
• background_image
  • database
  • service impl
• signature
  • database
  • service impl
• total_favorited
  • database
  • service impl
• work_count
  • database
  • service impl
• favorite_count
  • database
  • service impl

Integrate Wire DI to handle kitex clients dependencies

Add URL validation for mw.Auth to avoid illegal URL

Performance tests & efficiency optimization

So far, we have implemented only basic features and are not aware of their overall performance. Efficiency is critical in the development of business services. Thus, we should integrate OTel & run performance tests & chaos engineering to find bottlenecks in our system.

Unify protos and response fields.

Since we don't use protojson as the serializer anymore (due to the int64 serialization problem), We will need to unify & reconsider the response fields, make the IDLs fit with cloudwego sonic.

Sonic omits zero values by default, which causes the 0 status code to be omitted. The solution is to use optional, and the generated type would be *int. In this case, the zero value of the field would be nil or 0, and the 0 would be serialized.

Cleanup code

remove unsued imports and parameters etc.

Refactor all counting services.

Unfinished Feed Service

#4

There's still some TODOs in Feed service unfinished, fill them later.

请问您还留着简易版抖音apk吗？

请问您还留着简易版抖音apk吗？

Further performance optimization with caches & mqs

Performance: Feed function concurrency / bulk

Integrate ImageX as media file storage

Remove time field in the logger

Remove "time": time.Now() fields in the fields of the logger since it's already built-in.

Integration test.

Use testcontainers as the solution for databases & describe a series of operations that could validate if all the APIs are working properly.

Docking hitokoto API

From #29
https://developer.hitokoto.cn/

Use a distributed connection pool

[gossip]维持128star

已经129star了，我为了维持128又取消了star
现在又是128了，家人们我做的对吗

Standardize variable and function naming

• Use CountXXXRequest/Response instead of XXXXXCountRequest/Response in idl.
• Use actor_id and user_id instead of user_id or to_user_id
• ...

Wechat service bug fix

Integrate ImageX as video cover generator

Documentation

README.md && README_zh-CN.md

Use mw.GetAuthActorId(ctx)

Use mw.GetAuthActorId(ctx) in web gatewar handler when ActorID needed instead of call c.GetUInt32 directly.

Observability improvement