merzmax/scale-chat

scale-chat's Issues

Basic Server Monitoring

We want to be able to monitor some basic metrics using a Grafana dashboard.

Perform first load test

Introduce Workers

#6 did not fix all problems as expected.

Additionally the CSV file write, introduced with #26, does not complete writes when the loadtest client is getting terminated.
After some research, it turned out that the CloseConnection channel of the client didn't receive any items. Besides this there is a solution for the gracefull shutdown (close ws, close filewrite) of goroutines with the help of a waitGroup.

Here is an article that could help:
https://callistaenterprise.se/blogg/teknik/2019/10/05/go-worker-cancellation/
https://stackoverflow.com/questions/55535251/how-to-gracefully-shutdown-chained-goroutines-in-an-idiomatic-way

The goals of this issue are:

• understand how to use waitGroups
• create concept on how to terminate everything gracefully
• implement concept
• Write a small section about the issue → could get used for the end documentation

server - Implement global chat

The server should be able to receive and broadcast these messages to all clients that are connected to the server.

• Create new GO module that contians the models of the message that will be shared between the server and the client later on
• The server should be able to receive messages from connected clients and the messages sould be serialized
• The received messages should be sent to all connected clients

client - implement client that can chat wia the global chat

The client should be able to send/receive messages to/from the server's global chat.

• use the message model that was created in #9
• implement flag to the client that enables a manual chat
• if the flag is not set the client shoud send messages automatically
• the messagesize should be configurable by a flag
• the message frequency should be configurable to

Cannot run cAdvisor (on M1?)

I think we won't get around the fact that (at least I) need to run our test setup on a remote machine.
Unfortunately, cAdvisor crashes in my docker-compose deployment on my machine:

could not detect clock speed from output: "processor[...]

@MerzMax can you try if it crashes for you too? To run it, check out the branch feature/monitoring

In regards to the scaling options down the road, we may need to switch to Kubernetes at some point anyways. Maybe do it now?

scale-chat-cadvisor-1    | F1125 10:38:52.875927       1 cadvisor.go:175] Failed to create a manager: could not detect clock speed from output: "processor\t: 0\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 1\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 2\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 3\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\n"
scale-chat-cadvisor-1    | goroutine 1 [running]:
scale-chat-cadvisor-1    | k8s.io/klog/v2.stacks(0xc0000c4001, 0xc00018ce00, 0x5cc, 0x61c)
scale-chat-cadvisor-1    |      /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:996 +0xb8
scale-chat-cadvisor-1    | k8s.io/klog/v2.(*loggingT).output(0x2441740, 0xc000000003, 0x0, 0x0, 0xc00026a070, 0x2375833, 0xb, 0xaf, 0x0)
scale-chat-cadvisor-1    |      /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:945 +0x19d
scale-chat-cadvisor-1    | k8s.io/klog/v2.(*loggingT).printf(0x2441740, 0x3, 0x0, 0x0, 0x16bcdd4, 0x1e, 0xc000691ea0, 0x1, 0x1)
scale-chat-cadvisor-1    |      /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:733 +0x17a
scale-chat-cadvisor-1    | k8s.io/klog/v2.Fatalf(...)
scale-chat-cadvisor-1    |      /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1456
scale-chat-cadvisor-1    | main.main()
scale-chat-cadvisor-1    |      /go/src/github.com/google/cadvisor/cmd/cadvisor.go:175 +0x3b8
scale-chat-cadvisor-1    | 
scale-chat-cadvisor-1    | goroutine 3 [syscall]:
scale-chat-cadvisor-1    | os/signal.signal_recv(0x0)
scale-chat-cadvisor-1    |      /usr/lib/go/src/runtime/sigqueue.go:147 +0x9c
scale-chat-cadvisor-1    | os/signal.loop()
scale-chat-cadvisor-1    |      /usr/lib/go/src/os/signal/signal_unix.go:23 +0x22
scale-chat-cadvisor-1    | created by os/signal.init.0
scale-chat-cadvisor-1    |      /usr/lib/go/src/os/signal/signal_unix.go:29 +0x41
scale-chat-cadvisor-1    | 
scale-chat-cadvisor-1    | goroutine 4 [chan receive]:
scale-chat-cadvisor-1    | k8s.io/klog/v2.(*loggingT).flushDaemon(0x2441740)
scale-chat-cadvisor-1    |      /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1131 +0x8b
scale-chat-cadvisor-1    | created by k8s.io/klog/v2.init.0
scale-chat-cadvisor-1    |      /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:416 +0xd6
scale-chat-cadvisor-1    | 
scale-chat-cadvisor-1    | goroutine 9 [select]:
scale-chat-cadvisor-1    | go.opencensus.io/stats/view.(*worker).start(0xc000140100)
scale-chat-cadvisor-1    |      /go/pkg/mod/[email protected]/stats/view/worker.go:276 +0x100
scale-chat-cadvisor-1    | created by go.opencensus.io/stats/view.init.0
scale-chat-cadvisor-1    |      /go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x68
scale-chat-cadvisor-1 exited with code 255

Modify prometheus counter values

Modify prometheus message counter values in order to see if there are differences between the messages received by the clients and the ones received by the server.

Instead of one counter value it would be very intersting to see where the messages come from. In order to do this a new counter has to be implemented that displays the messages received by another server instance.

WebSocket Connection configuration

Check the configuration of the ws connection (configured in the server) to get familiar with at and improve configuration if it matches the needs of this project.

Process CSV files from loadtestclient

In order to use the collected data from the loadtest client a parser has to be implemented that calculaltes the following metrics:

• RTT
• percentiles for the message transfering time

Measure Time of a message roundtrip

Each client receives all messages, including the ones sent by himself. These messages can be used to save the RTT of a message

• Measure RTT
• Write RTT to console
• Toggle RTT measurement with a flag
• Save RTT to CSV?

Documentation for v1

To prepare the fist version, the README.md should contain all necessary information.

Close WS Connection gracefully

In order to decide if the disconnect of an client was intended or not, the intendet disconnect should be "quiet". At the moment the connection will not be closed gracefully.

If the client programm will be terminated the following output will be displayed in the server:

2021/11/05 18:41:10 Error during reading message:  websocket: close 1006 (abnormal closure): unexpected EOF

Close WS Connection gracefully

In order to decide if the disconnect of an client was intended or not, the intendet disconnect should be "quiet". At the moment the connection will not be closed gracefully.

If the client programm will be terminated the following output will be displayed in the server:

2021/11/05 18:41:10 Error during reading message:  websocket: close 1006 (abnormal closure): unexpected EOF

Client needs to be shut down when handler quits

If the receiveHandler returns, the client needs to shutdown itself and get removed from the load-test-client's client list.

Create simple chat

Based on the ws connection implementation, implement the possibility to send messages from one client to another one.

Add Server to Server comminication

If the server will be scaled, the different server instances will have to communicate with each other in order to distribute the messages to all clients. Therefore a new mechanism has to be introduced.

Add race detector

This video describes a feature / module that enables to detect race conditions when working with goroutines.

Tasks that should be fulfilled within this issue:

• Research about detectors
• Add a detector to the project

Containerize Server

The server application needs to be able to run in a Docker container.

Refactor the client management in the server

Put the client management part into a separate file.

See: #40

Incompleted CSV write

When the client is stopped the csv file will not be written as expected. There will be incompleted entries:

...
140,094160d2-d2b3-402e-96ee-ba4d79e40d7b,93c86b6e-0a56-4875-bba7-591a407d5b1b,Received,1639676670589293
140,e498c221-60e6-40f5-b202-ce6714a66c01,e498c221-60e6-40f5-b202-ce6714a66c01,Received,1639676670589309
140,8973851d-8a2b-4e13-9873-388d6d5090c5,2f07926e-12d3-47ba-9769-15a7b7d80c64,Received,1639676670589330
140,094160d2-d2b3-402e-96ee-ba4d79e40d7b,7983d741-643c-45de-afe5-1109bd3a0147,Received,1639676670589335
140,e498c221-60e6-40f5-b202-ce6714a66c01,8973851d-8a2b-4e13-9873-388d6d5090c5,Received,1639676670589346
140,094160d2-d2b3-402e-96ee-ba4d79e40d7b,2f07926e-12d3-47ba-9769-15a7b7d80c64,Received,1639676670589355
140,93c86b6e-0a56-4875-bba7-591a407d5b1b,e498c221-60e6-40f5-b202-ce6714a66c01,Received,1639676670589378
140,2f07926e-12d3-47ba-9769-15a7b7d80c64,34eaf832-3187-4de9-be5c-7d74e8e12558,Received,1639676670589594
140,e498c221-60e6-40f5-b202-ce6714a66c01,34eaf832-3187-4de9-be5c-7d74e8e12558,Received,1639676670589669
140,8973851d-8a2b-4e13-9873-388d6d5090c5,fc315ebf-3b34-4

Grafana dashboard panels for cAdvisor

Create panels for data from cAdvisor

Basic web client

For demonstration purposes we should have a basic websocket client for the browser that is capable of communicating with our server.

Establish a basic WS connection

• The server should have an endpoint /wsthat can be used to establish a websockets connection.
• The client should be able to connect to the server with websockets

Limit server resources to simplify load tests

The server container should only have small resources so that it fails quicker.

Chat Rooms in Server

Research and implement Chat-Rooms in the server

Remove dead clients on server

• ws was cloes by client
• ws connection errors in general
• error while reading from the ws connection