codex-team/hawk.collector

High-performance messages collector for the Hawk project

hawk.collector's Issues

Verify that manual queue forging is impossible

Create a whitelist of possible queue names.

Configuration file

Load all parameters from config.json

• RabbitMQ connection URL with credentials
• connection retry limit
• exchange and rabbitMQ settings (persistence, autoDelete, ack etc.)
• listen host and port
• exchange name
• default path

Add config generation wizard if not found.

Update linter

gometalinter is deprecated. Update to GolangCI-lint

Setup CI

Deploy scripts

Build scripts to setup

• RabbitMQ
• Nginx with load balancing
• Golang server

Write example of release sending to collector

Implement logging of redis cache updates

Logs should include message Banned projects list have been updated. Number of projects size changed 123 → 1245

Collector must close gracefully when rabbitmq is not available

Now it just panics

2021/04/17 13:35:37 Failed to connect to RabbitMQ (amqp://guest:guest@rabbitmq:5672/): dial tcp: lookup rabbitmq: no such host
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x13523bb]

Add makefiles for cross-platform builds

Check how it is resolved in other projects

Write test plan

Add testings docs
Add integration testing guide
Add samples and typical commands

RabbitMQ definitions and topology deploy script

Build script and define project topology in a config.
Provide auto-deploy the topology in docker

https://www.rabbitmq.com/backup.html#rabbitmq-definitions
https://stackoverflow.com/questions/40280293/defining-a-queue-with-a-config-file-in-rabbitmq

Error during hawk.Catch with disabled catcher in ENV

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x737c96]

goroutine 135 [running]:
github.com/codex-team/hawk%2ego.(*Catcher).Catch(0x0, 0xabd040, 0xe61d20, 0xc000132cb0, 0x19)
	/go/pkg/mod/github.com/codex-team/[email protected]/catch.go:89 +0xe6
github.com/codex-team/hawk.collector/pkg/hawk.Catch(0xabd040, 0xe61d20)
	/build/pkg/hawk/main.go:48 +0x4e
github.com/codex-team/hawk.collector/pkg/server.(*Server).handler.func1(0xc00010fc70, 0xc000183200)
	/build/pkg/server/server.go:88 +0x131
panic(0x960d00, 0xe61d20)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/codex-team/hawk%2ego.(*Catcher).Catch(0x0, 0xabc920, 0xc00004fc10, 0xc000369b68, 0x40c7a8)
	/go/pkg/mod/github.com/codex-team/[email protected]/catch.go:89 +0xe6
github.com/codex-team/hawk.collector/pkg/hawk.Catch(0xabc920, 0xc00004fc10)
	/build/pkg/hawk/main.go:48 +0x4e
github.com/codex-team/hawk.collector/pkg/server/errorshandler.(*Handler).HandleWebsocket(0xc000081728, 0xc000183200)
	/build/pkg/server/errorshandler/handler_websocket.go:46 +0xa6
github.com/codex-team/hawk.collector/pkg/server.(*Server).handler(0xc000081680, 0xc000183200)
	/build/pkg/server/server.go:97 +0x1f1
github.com/valyala/fasthttp.(*Server).serveConn(0xc0000fa200, 0xacbd40, 0xc00000e1b0, 0x0, 0x0)
	/go/pkg/mod/github.com/valyala/[email protected]/server.go:1926 +0x878
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc0001275e0, 0xc000227820)
	/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:212 +0xc0
github.com/valyala/fasthttp.(*workerPool).getCh.func1(0xc0001275e0, 0xc000227820, 0x91c1a0, 0xc000227820)
	/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:184 +0x35
created by github.com/valyala/fasthttp.(*workerPool).getCh
	/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:183 +0x10a

Add ACK subscription to the protocol

The idea is to provide token for user subscription.
He can use this token to connect to the pub-sub server (centrifugo) and get information about processings statuses or other statistics

Empty payload causes 500

Input
{ "token": "eyJhbGci<...>iJN0dg", "catcherType": "errors/python" }

Result
{ "error": true, "message": "Server error", "status": 500 }

Make `commits` fields in release optional. As long as file

Setup docker compose config for production

RabbitMQ reconnect research

Check if connection should check failovers

RabbitMQ reconnect

Check reconnect during work feature.
Add proper logging events for connection issues.

Add a release route. Refactor a sourcemap route.

You need to create a new route /release that will accept the release data and will include the source maps.
https://docs.codex.so/suspected-commits#izmeneniya-v-kollektore

Support multi stage build for Golang

Build tiny image as in the docs:

https://docs.docker.com/develop/develop-images/multistage-build/#use-multi-stage-builds

https://medium.com/travis-on-docker/multi-stage-docker-builds-for-creating-tiny-go-images-e0e1867efe5a

metrics storage

Documentation and description of a common communication format

Parallel research

Compare if there can be a significant difference in performance when use multiprocessing and different number of connections.

Rename catcher_type to catcherType

Statistics

Provide variants of runtime monitoring

• Number of connections per user
• Messages rate per user

Golang 1.13 compatibility

Review and use new features (especially Error Inspection)

Govendor

Check if it is possible to remove vendor files and add govendor fetch to the deploy scripts and docker

JWT support

Parse JWT and check validity of digital signature

Create ability for blocking the particular IP address

EasyJSON research and deploy

Compare JSON marshalling and unmarshalling speed during highload for EasyJSON and standard JSON in golang.

If easyJSON is better – generate helpers

json-iterator research

Benchmark json-iterator/go

Migrate from vendor to go modules (go 1.11)

Decide if this is stable enough:
https://github.com/golang/go/wiki/Modules
https://go.googlesource.com/proposal/+/master/design/24301-versioned-go.md

Implement event batching support

Add the ability to receive array of events

Release-commits structure

Commits sent during release must follow the structure (or be omitted):
– hash
– title
– author
– date

Websockets

Support error messages delivery through websockets

Travis CI

Add travis and write integration tests

Detect suspicious activity

JSON config beautifulize

Config must be human-readable:

{
  "broker_url": "amqp://guest:guest@localhost:5672/",
  "exchange_name": "errors",
  "server_host": "localhost",
  "server_port": 3000,
  "retry_number": 5,
  "retry_interval": 3
}

The use of standard MarshalIndent feature would be appropriate.