p-society/gc-server

Stay updated in real-time and engage with the thrill of the game like never before.[WIP]

GC Score Broadcasting Service

Grand Championship Broadcasting System seamlessly bridges web and app platforms, improving the experience of our annual sports championship. Stay updated in real-time and engage with the thrill of the game like never before.

P-Society Handles

Website | Community Discord | Contribute

---

GCSB / Grand Championship Broadcasting Service is a live score broadcasting application for both web and app platforms. The application is built using Golang,Kafka,MongoDB, SSE (Server-Sent Events), and Redis. The web version is being developed using Next.js, while the app version is built using Flutter.

GCSB offers two primary benefits to its users.

• Real-time score updates for various sports events.
• Integration with web and app platforms.
• Utilization of Golang and WS for data transfer.
• Data storage and retrieval using MongoDB and Redis.

---

Current contributors

Made with contributors-img.

License

GC-Server and GC-Client is licensed under Apache License

gc-server's Issues

[FEAT]: Create Dockerfiles and Compose files for Orchestrations

To ensure consistency across environments, we aim to containerize our Auth , Mail , and Core APIs using Docker. This involves creating Dockerfiles for each service and orchestrating them with a Docker Compose configuration.

Auth API: Develop a Dockerfile for our Auth API service, ensuring it includes all necessary dependencies and configurations.
Mail API: Construct a Dockerfile for our Mail API service, considering any specific requirements it may have.
Core API: Build a Dockerfile for our Core API service, including any libraries or tools required for its functionality.
Develop a docker-compose.yml file to orchestrate the deployment and interaction of the Auth, Mail, and Core APIs.
Define service configurations within the Docker Compose file, specifying dependencies, ports, environment variables, and any other   necessary settings

Ensure proper networking and communication between the services within the Docker Compose environment.

Document the Dockerfile structure and usage guidelines for each service to facilitate future development and maintenance.
Provide instructions on how to use the Docker Compose configuration for local development, testing, and production deployment.

[DISCUSSION] : Consideration of Using Kafka for Streaming Data to Clients

The goal of the GCSB project is to develop a robust system with multiple independent and decoupled APIs for sports. Currently, Server-Sent Events (SSE) have been identified as a suitable choice for achieving real-time comm. from server-->client due to their lightweight nature and ease of setup,but there are certain concerns that needs to be solved.

Key Concerns:

1. API Design Uniformity:
   How should we design the APIs to ensure uniformity across the project? Should we opt for a single SSE or individual SSE for each API ?
   We are aiming for a uniform approach that can enhance consistency and ease maintenance.

• Single vs. Individual SSE: Trade-offs between having a single SSE for all APIs or individual SSE for each API.
• Design Principles: We need to brainstorm on design principles to be followed for API endpoints, naming conventions, and response formats.

2. Need for Kafka/Similar Queue or Pub-Sub Messaging:

   Question: Considering a maximum of 1000 concurrent users at worst case, do we really need a queuing or pub-sub architecture like Kafka/RabbitMQ? What are the pros and cons? Can SSE alone can handle the expected load or if a more scalable solution is necessary?

Please share your thoughts, concerns, and suggestions regarding the API design uniformity and the need for a queuing/pub-sub architecture in this issue thread.

Consider this as a High Priority Issue.

[DOCS] End-to-end documentation

Project requires a good guide on several topics so that new developers can contribute to the project

• Create a good README file that includes project information, installation instructions, usage guidelines, and contribution guidelines.
• Keep an eye on the repo and regularly update the README file to reflect changes in the project, new features, or updates.
• Create Changelog file establish a changelog file to document changes made in each release.(Low priority for now).
• Developer Documentation (Guide + Setup).Document the project's code structure and organization to help developers navigate the codebase.
• Contribution Guidelines
• Contribution Process
• Document the process for contributing to the project, including how to submit bug reports, feature requests, and pull requests. Provide a code style guide to maintain consistency in the codebase.

[FEAT]: UI,Frontend for Mobile Applications and Web

Currently,The server code is in active development alongside testing phases. Fundamental API endpoints are being developed, setting the groundwork we envision.We are also currently testing the many options we have in the server side development,and it would take some time to reach it's fulfillment.

Concurrently, we shall gear up for the design and development phase of client-side interfaces, for both mobile apps and web interfaces. Integration between front-end and back-end components is crucial for the success of our project and needs collective efforts.🇷🇺

• The design phase for client-side interfaces encompassing wire frames, user flows, and interface mock ups.
• Start the development of client-side applications, ensuring exact integration with services.

This thread will serve as our central hub for discussions concerning the synchronization of front-end and back-end development. Kindly utilize this thread as a PARENT THREAD for all related discussions, keeping the communication streamlined and accessible.

Your insights, thoughts, and recommendations on the design and development process are highly valued.

[DISCUSSION] Usage of `GridFS` for Multimedia File Uploads all across the project

We shall be using MongoDB GridFS for handling multimedia file uploads in our project instead of s3 or blob storage , as the multimedia part is limited (eg:Player Image,etc..) .GridFS is easy to work with (+Cheaper$$).

Open to discussions and feedback regarding the proposed solution.

[FEAT]: (Client) Update contents of title in web apps

For instances of the application running on the browser, display the score in the 'title' of the page and update it as changes occur.

[PROPOSAL]: Unit Testing and TDD Practices

As our team works on the project, we should recognize the importance of establishing a testing environment to ensure the reliability and maintainability of our codebase. This issue proposes the implementation of unit testing practices and adopting Test-Driven Development (TDD) as part of our development process.

I know it will slow down our Development speed and maybe is a redundant thing,but incorporating TDD from the beginning instills good development practices.

Open for suggestions

CI Implementation for Go and Node.js Servers

The goal is to implement Continuous Integration (CI) for the project by using TravisCI / Github Actions (preferred) , which which currently has some Go and Node.js servers (planned). The project structure includes some Go servers and Node.js servers in the src/api directory.
The current CI is flaky and unreliable.

As the project is now in it's initial stages,lets keep the CI simple - keeping it limited to build and lint checks for the above mentioned languages only.

[FEAT] : Core feature API for PSA, Admin and Players.

APIs to be implemented:

Player:

• Update : PATCH /v0/players
• Delete (soft): DELETE /v0/players

PSA Upgrades player to admin:

Have separate collections for Admin/Player and PSA
PSA can have same mail which was used to register as player.

• Upgrade: PATCH /v0/players/upgrade:id

After a successful upgrade:

1. Send mail to Admin to notify the upgrade and the reason
2. Send app notification to user about the same

Admin is responsible for moderating a match and all required API endpoints for that will be described in a separate thread.

PSA creates a Match and assigns an Admin to it:

• Create Match: POST /v0/matches
• Delete Match (soft): DELETE /v0/matches:id
• Update Match : PATCH /v0/matches

The PSA creates a match with the following details:

1. Squads/Teams/Branches Facing off
2. Admin moderating it
3. Venue
4. Type
5. Sport
6. Time and Date
7. Stage of match

PSA creates squads for the tournament:

A squad is an independent participant of the tournament (e.g. CSE-2 , IT-1 etc) and is a super set of teams that will play matches.
PSA will select a set of players from all verified players who have registered to the platform for a squad.

• Fetch all players: GET /v0/players/
• Create squad: POST /v0/squads

Include details:

1. Squad name (Branch name + /1/2)
2. Squad members
3. Other required details

Entity Relationships:

Event, Team, Player, Match, Stats and Squad are separate entities.

• There are two collection mappings which is squad player and team player.
• Squad has squad-player, player is a squad player, squad player is in a team, team has team players, team player derives from squad player , squad plays match, team plays match, admin moderates match, match generates events.

Creating Teams

After PSA assigns a match to an Admin, the admin is responsible to create the team that plays in that match.
Match is a collection that has admin id, squad 1 and 2 id, team 1 and 2 id.

Admin can create teams from 'squad-player' collection

• Fetch squad-players: GET /v0/players/
• Create Team for match: POST /v0/teams/

Events

A match generates events
Events has team players + event data + sport + user id+ match id + tbd
A single player will have a single stat collection, stat document for the user will be updated after each event
Events will be trailed to the client in real-time + stored in db + tbd
Events will be the core

To be further discussed: Events implementation, Statistics Calculation, Substitutions.

[FEAT] : Implementation of event streaming through Kafka

Events are any real world match instances which contain a snapshot of the match. For instance, in cricket: a ball, a wicket, a substitution, an interruption by nature, a new batsman entering the crease etc are all events.

This particular service will be responsible for accepting the data from client (admin), processing it (if required) and pushing it to other layers of services managing SSE connections to the clients (viewers).

Need for Kafka:

• Ensuring no data is lost (consistent writes to DB )and the system is fault tolerant
• Managing dead letter queues to ensure chronological delivery of events
• Synchronising multiple microservices

Extras to be implemented:

• Dedicated logger for Kafka
• Unit and Integration testing

All implementations are to be done using the go library sarama

Refer past discussions regarding usage of Kafka and Sarama: #4

All queries and suggestions related to the event streaming service(s) can be done here.

🎉 Trello work board for organising workflow of the project 🎉

A Trello work board has been created to assign, track and review the work being done. Contributors are advised to join the board to ensure an organised flow of work.
Click here to view the board.

[FEAT]: Basic actions required for admin end of application

• Register : Register players, teams and referees/umpires, and store them in database so they can be made available while creating games.
• Creating games: The scorers can create a new game which takes in players, teams and game orientations. New players can be added on spot. Games can be paused and deleted.
• Scoring games: The games can be scored by sending 'events' which are a snapshot of the game a particular instance containing all parameters for the set orientation. Events can be independently stored for each game to view the game and undo events.
• Update standings: The standings can be manually updated to cater for unavoidable circumstances.

Please list down other necessary features and suggest API endpoints to implement each feature in this thread.

[FEAT]: Creation of ML model regarding win prediction of our GC cricket tournament.

So the idea is that we have seen the win probability of google during the IPL and international matches. How about we create a same system but its for our own college and GC!

[FEAT]: Prevent opening in multiple tabs

A functionality can be built to prevent the app from being opened in multiple tabs like WhatsApp web does. This is to cater the limitation that SSE can have only 6 concurrent tabs open at a time and also reduce connections per user with the server.

[FEAT]: Implementation of Player APIs

We have developed a player API to support player dashboards. However, there are several functionalities and enhancements that need to be implemented to improve the overall functionality.

Read All Players API:

Implement an API endpoint to retrieve all player data.
Ensure proper pagination to handle large datasets.
Include flags or filters to allow for customization of the query results.

Update All Players API:

Develop an API endpoint to update information for multiple players simultaneously.
Implement validation checks to ensure data integrity and security.
Provide options for bulk updates with proper error handling.

Soft Deletion of Players API:

Create an API endpoint to perform soft deletion of player records.
A flag or status field to indicate deleted players without removing them from the database to be implemented.
Ensure proper handling of soft deletion.

Enhancements:

Proper pagination: Implement pagination for large datasets to improve performance and user experience.
Flags and filters: Allow users to customize queries with flags and filters to retrieve specific subsets of data.
Validation checks: Implement validation rules to ensure data integrity and prevent invalid updates.
Error handling: Provide comprehensive error handling to handle various scenarios gracefully and provide meaningful feedback to users.
Security considerations: Implement authentication and authorization mechanisms to restrict access to sensitive endpoints and data.
Documentation: Document APIs and endpoints thoroughly to facilitate integration and usage by developers.

[DISCUSSION]: Server Side Events or Websockets?

In light of recent discussions and insights, I'm reconsidering our decision to utilize SSE over Websockets for emitting scores to clients.

During a conversation with a senior colleague, they shared their experience building a product (Hubilo Studio) using Websockets, WebRTC,and a LOT more things, - capable of scaling and handling up to 1L~ concurrent users. This has prompted me to reassess our choice.

He emphasized that Websockets currently offer the most robust solution, particularly considering the ecosystem and industry trends. He highlighted that many companies are shifting towards Websockets for real-time communication due to the comprehensive support and scalability options available and OSS tools around it

Our initial concern (as we , @majorbruteforce and @me discussed ) was scalability for the GC server if we choosed ws. If viable solutions exist for Websockets, it seems prudent to explore them further. The broader support and resources available for it compared to SSE may offer advantages in terms of implementation and future maintenance.

I propose revisiting our technology stack decision and considering the potential benefits of adopting Websockets for our real-time communication needs,as most of the work in the core sections (majorly the crud apis,are completed).
Stuffs like auto-healing, handling backpressure, message sharing between distributed servers using redis, connection retry mechanisms, and ack/nack support might align better with our requirements and facilitate smoother integration and maintenance.

Feedback and opinions on this matter are welcome for further discussion.
CC: @majorbruteforce @punitkr03 @soumil-kumar17

[TRIAL]: MVP Mobile App to check compatibility of SSE Servers

We have checked compatibility + working of SSE Servers in client side for the web.
We need to develop a rough application for trial purposes to check the ability to capture EventSource (Server-Sent Events or SSE) in Mobile Apps in order to enable real-time updates in client side. This will serve as a crucial step in exploring and implementing real-time communication within our application.

It's just for trial,so no need to have a design,just a simple flutter app that consumes data from a simple SSE server.
One can check out Eventsource and http package to implement the same.

Anyone from backend team can assist you in creating a sample SSE Server ,providing you with a SSE Endpoint.

Articles you might find useful :
What is SSE : https://ably.com/topic/server-sent-events
Official eventsource package at pub.dev : https://pub.dev/packages/eventsource
SSE + Flutter : https://medium.com/flutter-community/server-sent-events-sse-with-flutter-cf331f978b4f

Please Keep in mind,this is NOT Websockets ,This is SSE.

Steps for Development :

• Create a branch app:sse-trial from main branch
• All the work related to server and App to be done there.
• As it's a trial , it Won't be merged into :main

[FEAT]: Implementation for PSA Routes