anycable/anycable-go

AnyCable real-time server

AnyCable-Go WebSocket Server

WebSocket server for AnyCable.

AnyCable Pro has been launched 🚀

Versioning

Important Use the same major version of AnyCable-Go as your AnyCable gem. AnyCable-Go uses the same major version number (and minor version as well for 0.x series) as other libraries to indicate the compatibility.

Installation

The easiest way to install AnyCable-Go is to download a pre-compiled binary:

# Example for `anycable-go-linux-amd64`
curl -fsSL https://github.com/anycable/anycable-go/releases/latest/download/anycable-go-linux-amd64 -o anycable-go
chmod +x anycable-go
./anycable-go -v

MacOS users could install it with Homebrew

brew install anycable-go

Arch Linux users can install anycable-go package from AUR.

Of course, you can install it from source too:

go install github.com/anycable/anycable-go/cmd/anycable-go@latest

For JavaScript projects, there is also an option to install AnyCable-Go via NPM:

npm install --save-dev @anycable/anycable-go
pnpm install --save-dev @anycable/anycable-go
yarn add --dev @anycable/anycable-go

# and run as follows
npx anycable-go

Upgrade

For instructions on how to upgrade to a newer version see upgrade notes.

Heroku

See heroku-anycable-go buildpack.

Usage

Run server:

$ anycable-go

=> INFO 2020-02-05T08:44:57.684Z context=main Starting AnyCable 1.1.0

You can also provide configuration parameters through the corresponding environment variables (i.e. ANYCABLE_RPC_HOST, ANYCABLE_REDIS_URL, etc).

For more information about available options run anycable-go -h.

📑 Documentation

Build

# first, prepare mruby (we embed it by default)
# NOTE: Might require running with sudo, since we build artifacts within a Go module
make prepare-mruby

# then build the Go binary (will be available in dist/anycable-go)
make

You can run tests with the following commands:

# Run Golang unit tests
make test

# Run once
make prepare

# Run integrations tests
make test-conformance

# Run integration benchmarks
go install github.com/anycable/websocket-bench@latest
make benchmarks

We use golangci-lint to lint Go source code:

make lint

Docker

See available images here.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/anycable/anycable-go.

Please, provide reproduction script (using this template) when submitting bugs if possible.

License

The library is available as open source under the terms of the MIT License.

Security Contact

To report a security vulnerability, please contact us at [email protected]. We will coordinate the fix and disclosure.

anycable-go's Issues

Links to download binaries are not working

When I try to download a pre-compiled binary it is giving me permission denied
Tried Linux x86_64.

[]s

Client 502

AnyCable-Go version: 0.6.0-preview5
AnyCable-rails gem version: 0.5.4
AnyCable gem version: 0.5.1
gRPC gem version: 1.12.0

Приветствую, @palkan!

AnyCable-Go log:

D 2018-07-03T08:34:21.815Z context=rpc Authenticate response: status:FAILURE 
D 2018-07-03T08:34:21.815Z context=ws Websocket session initialization failed: Application error: 
D 2018-07-03T08:34:21.836Z context=rpc Authenticate response: status:FAILURE 
D 2018-07-03T08:34:21.836Z context=ws Websocket session initialization failed: Application error: 
D 2018-07-03T08:34:21.837Z context=node sid=0jEsitBFLew~1PUSRQPRnI Websocket read error: websocket: close 1000 (normal)

...
D 2018-07-03T08:34:27.632Z context=node sid=1PsgHl_1F4zfUGVjECi_n9 Perform result: &{[] false [] false}
D 2018-07-03T08:34:27.789Z context=node sid=0bVugylOYekDWweTqVQbBM Websocket read error: websocket: close 1006 (abnormal closure): unexpected EOF
D 2018-07-03T08:34:27.789Z context=node sid=0bVugylOYekDWweTqVQbBM websocket session completed

В логах AnyCable ошибок нет, кроме reject.

Метрики:
Client code 101:

I 2018-07-03T08:37:55.309Z broadcast_msg_total=1 broadcast_streams_num=240 client_msg_total=85 clients_num=242 clients_uniq_num=239 context=metrics disconnect_queue_size=0 failed_auths_total=121 failed_broadcast_msg_total=0 failed_client_msg_total=0 goroutines_num=745 rpc_call_total=206 rpc_error_total=0

Client code 502:

I 2018-07-03T08:40:55.324Z broadcast_msg_total=0 broadcast_streams_num=235 client_msg_total=0 clients_num=237 clients_uniq_num=234 context=metrics disconnect_queue_size=4 failed_auths_total=0 failed_broadcast_msg_total=0 failed_client_msg_total=0 goroutines_num=1586 rpc_call_total=2 rpc_error_total=0

Помогает только перезагрузка RPC сервера
Буду благодарен любым комментариям

ERROR on installing from source

go get -u -f github.com/anycable/anycable-go/cmd/anycable-go

Delivers ERROR:
package context: unrecognized import path "context" (import path does not begin with hostname)

Windows binaries should have .exe suffix

anycable-go-win-386 → anycable-go-win-386.exe

Redis Sentinel set passwords

Hi there!

For anycable-go 1.0.0, how can I set/pass a password for the Sentinels in ENV var

ANYCABLE_REDIS_SENTINELS=127.0.0.1:26380,127.0.0.1:26381

or is this maybe not possible yet?

Thank you in advance
Martin

No documentation on Docker usage

It would be nice if there were environment variables and basic usage available in Docker Hub or in the README.

Looking to implement ticket-based authentication using client IP address

I am trying to implement an authentication method using tickets (described here), but this is impossible to achieve without anycable-go proxying the websocket client ip address to the gRPC server. Is there a way to achieve this?

Проблемы c anycable-go на AWS

Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

**AnyCable-Go version: **
https://github.com/anycable/anycable-go/releases/download/v0.6.2/anycable-go-v0.6.2-linux-amd64

AnyCable gem version: doesn't matter

Наблюдаю на AWS такую вот странную штуку. -v, -h и пр:

/home/ubuntu/gocode/bin/anycable-go: line 1: syntax error near unexpected token `<'
/home/ubuntu/gocode/bin/anycable-go: line 1: `<html><body>You are being <a href="https://github-production-release-asset-2e65be.s3.amazonaws.com/67314608/b61b7c80-4f0b-11e9-9d3f-ea23380f0676?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190609%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20190609T183919Z&amp;X-Amz-Expires=300&amp;X-Amz-Signature=055a4300e1ba26473af04b13a5376917dcfe34eeec8e8544e7196d90ef28b5f1&amp;X-Amz-SignedHeaders=host&amp;actor_id=0&amp;response-content-disposition=attachment%3B%20filename%3Danycable-go-v0.6.2-linux-amd64&amp;response-content-type=application%2Foctet-stream">redirected</a>.</body></html>'

возможно не ту версию поставил.
при пересборке из сорсов все выводит норм.

Stability of the rpc instance

вопрос: как быть с автохилом rpc? в случае когда у нас это рельсы под пумой, там проблем нет, пума их как бы лечит сама, но rpc инстанс сам по себе, и если он ляжет то вся система сломается.

И еще момент, хелс респонс от anycable-go не учитывает, доступность для него rpc, если rpc по каким-то причинам недоступен для anycable-go, и это не будет отражено в хелс респонсе.

Typo? "anycable-go-mrb-dawrin-amd64"

dawrin probably should be darwin

SSL Support

Could you add built-in SSL support?

Add istate to Disconnect calls

Related to https://github.com/anycable/anycable/issues/108

Redis subscription stops working

Hey!

I noticed one issue with the server: after 10 hours of running the anycable-go it stopped receiving Redis messages with broadcasts.

I inspected logs using -log option and manually subscribed to the Redis channel via redis-cli and the messages are coming, but anycable-go does not seem to receive them.

My best guess is that it does not correctly reconnect after timeout/network issues.

Restarting the anycable-go fixes the issue, but temporarily until another issue where it stops receiving pubsub from Redis again.

Some artificial steps to reproduce:

1. Start WS subscription the usual way
2. Perform broadcasts to see that it works
3. Stop Redis server
4. Start Redis server
5. Anycable-Go never tries to connect again and further PUBSUB messages from Redis are lost and never arrive again

Broadcast string appears as null

Thanks for an awesome project! I would love to try this in our production app. I think we can save a lot of heroku dynos as we have problem with memory and number of connections.

We are sending server rendered html or json returned by postgres directly with ActionCable, but it do look like this isn't working with AnyCable.

# Works
ActionCable.server.broadcast "class_section_189", { name: "Testing" }

# Does not work
ActionCable.server.broadcast "class_section_189", "Testing"

Versions
Rails 5.1.4
Ruby 2.4.2
AnyCable websocket server v0.5.0.1

Docker Install Error

Tell us about your environment

AnyCable-Go version:
1.0.0rc1

AnyCable gem version:
1.0.0rc1

What did you do?

Install anycable-go via Docker

What did you expect to happen?

Successfully install

What actually happened?

2020-06-14T12:54:30.127Z: #9 83.45 Fetching grpc 1.28.0
2020-06-14T12:54:30.792Z: #9 83.92 Installing grpc 1.28.0 with native extensions
2020-06-14T13:00:21.366Z: /home/bas/rubydeployer/scripts/build-docker.sh: line 20:   749 Killed                  DOCKER_BUILDKIT=1 docker build "${env_opt[@]}" -f "${dockerfile}" -t clever --rm=true .
2020-06-14T13:00:21.366Z: Error while building Docker. Aborting. Please check the documentation about Docker configuration: https://www.clever-cloud.com/doc/docker/docker/
2020-06-14T13:00:21.366Z: Deploy failed
2020-06-14T13:00:21.540Z: Deploy failed in 7 minutes and 31 seconds. Please review the lines above to find out why
2020-06-14T13:00:21.625Z: time="2020-06-14T13:00:21.280763176Z" level=warning msg="grpc: addrConn.createTransport failed to connect to { 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc
2020-06-14T13:00:21.933Z: time="2020-06-14T13:00:21.689350600Z" level=error msg="failed to kill runc moeg16zcz1w6eceul4clajaqh: runc did not terminate sucessfully: container not running\n"
2020-06-14T13:00:27.040Z: Time to clean up and say good bye.
2020-06-14T13:00:27.125Z: Good bye!

And anothers build's Out of Memory occured
Config is 1Gb Memory and 1 CPU

2020-06-14T09:38:02-03:00 #9 108.6 Fetching grpc 1.28.0
2020-06-14T09:38:03-03:00 #9 109.2 Installing grpc 1.28.0 with native extensions
2020-06-14T09:43:45-03:00 /home/bas/rubydeployer/scripts/build-docker.sh: line 20: 628 Killed DOCKER_BUILDKIT=1 docker build "${env_opt[@]}" -f "${dockerfile}" -t clever --rm=true .
2020-06-14T09:43:45-03:00 An Out of Memory occured. If it was during a build, enable the "Enable dedicated build instance" in the application's page.
2020-06-14T09:43:45-03:00 If it was during runtime, it may be because you use a scaler that doesn't have enough memory or a memory leak happened.
2020-06-14T09:43:45-03:00 You can select a bigger scaler in the "Scalability" page of your application
2020-06-14T09:43:45-03:00 Error while building Docker. Aborting. Please check the documentation about Docker configuration: https://www.clever-cloud.com/doc/docker/docker/

Panic in (*Session).Send()

panic: send on closed channel 
goroutine 19 [running]: 
github.com/anycable/anycable-go/node.(*Session).Send(0x1a3f5e00, 0x1b99c000, 0x6e81e, 0xa5bdd) 
/Users/palkan/dev/golang/src/github.com/anycable/anycable-go/node/session.go:149 +0x48 
github.com/anycable/anycable-go/node.(*Hub).broadcastToStream(0x19ba22a0, 0x1b1f0bf0, 0xe, 0x1b804000, 0x6e7d4) 
/Users/palkan/dev/golang/src/github.com/anycable/anycable-go/node/hub.go:248 +0x2a1 
github.com/anycable/anycable-go/node.(*Hub).Run(0x19ba22a0) 
/Users/palkan/dev/golang/src/github.com/anycable/anycable-go/node/hub.go:105 +0x1e5 
created by github.com/anycable/anycable-go/node.NewNode 
/Users/palkan/dev/golang/src/github.com/anycable/anycable-go/node/node.go:90 +0x14c 

AnyCable-Go version: 0.6.0-preview4
AnyCable gem version: 0.5.0

Trouble installing on linux (ec2)

go get -u -f github.com/anycable/anycable-go/cmd/anycable-go returns # github.com/anycable/anycable-go/metrics /root/go/src/github.com/anycable/anycable-go/metrics/prometheus.go:16:10: undefined: strings.Builder

How to deploy ssl websocket with anycable?

Hi, I'm using Puma, Nginx and anycable-go, How can I deploy wss server? thanks.

Example systemd file

Are there any example systemd files available? Or best practice examples of running on ubuntu (16.04), I'm struggling to get the server running when using systemd.

Thanks

Сообщение доставлено?

есть какой-то более лаконичный способ навскидку получения информации о доставленности/недоставленности сообщения на клиента? Кроме подтверждения с клиента через вызов самописной функции.

Из кода я так понял, что сбой при отправке сообщения не нотифицируется никуда, получается что если произошел сбой при отправке на клиента, то сообщение из очереди редиса просто исчезнет, а сам код рубишный об этом не узнает?

Healthcheck

Hi,

Are there any plans to implement a healthcheck for the anycable-go websocket server similar to the anycable gem? We are setting this up behind an AWS application load balancer which only supports http healthchecks.

Thanks

connection stress test outcomes

I would like to ask what is the test figures for these outcomes

1. connection with 1000 concurrent connections with open 65540 open files and unlimited processes
2. operate over 10 hours
3. see any race detections?

Make "Auth Failed" easier to debug

Thank you @palkan for this wonderful gem.

I'm trying to deploy Anycable with Anycable-go using Dokku (docker-based deployment scripts utilizing Heroku buildpacks). While Anycable-go worked for me out of box on localhost, I'm having troubles making it work on production, behind Nginx.

When I run anycable-go using
/app/bin/anycable-go/anycable-go-0.4.1-linux-amd64 -redis=[redacted] -rpc=localhost:50051 -addr=0.0.0.0:8080 -log=true

I get

Ping interval 3s
Running AnyCable websocket server v0.4.1 on 0.0.0.0:8080 at /cable
Disconnect rate 10ms
anycable: subscribe 1

When the first request comes in (through Nginx), I see

Auth
Auth Failed

Can we please make this error message more descriptive and/or include tips for troubleshooting? I've tried running the RPC on 0.0.0.0:50051 as well as setting SKIP_AUTH=1 in environmental variable.

I'll be grateful for any suggestions. Thank you.

A strange behavior when running anycable-go (systemd restarts)

I've just published my system but sometimes systemd restarts this server throwing this error:

Sep 13 03:08:29 debian-yggdrasil bash[5523]: Broadcast to stream sensor_1: {"user":"external","message":"22.97","sid":98016}
Sep 13 03:08:29 debian-yggdrasil bash[5523]: panic: send on closed channel
Sep 13 03:08:29 debian-yggdrasil bash[5523]: goroutine 8 [running]:
Sep 13 03:08:29 debian-yggdrasil bash[5523]: main.(*Hub).run(0xa99d40)
Sep 13 03:08:29 debian-yggdrasil bash[5523]: #011/Users/palkan/lib/go/src/github.com/anycable/anycable-go/hub.go:107 +0x9f8
Sep 13 03:08:29 debian-yggdrasil bash[5523]: created by main.main
Sep 13 03:08:29 debian-yggdrasil bash[5523]: #011/Users/palkan/lib/go/src/github.com/anycable/anycable-go/server.go:176 +0x281
Sep 13 03:08:29 debian-yggdrasil systemd[1]: anycable-go.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 13 03:08:29 debian-yggdrasil systemd[1]: anycable-go.service: Unit entered failed state.
Sep 13 03:08:29 debian-yggdrasil systemd[1]: anycable-go.service: Failed with result 'exit-code'.
Sep 13 03:08:39 debian-yggdrasil systemd[1]: anycable-go.service: Service hold-off time over, scheduling restart.

AnyCable version 1.0 not installing successfully

Tell us about your environment

golang 1.14

AnyCable-Go version:
1.0

AnyCable gem version:
0.6

What did you do?

tried to install using go get -u -f github.com/anycable/anycable-go/cmd/anycable-go

What did you expect to happen?

to install and run successfully

What actually happened?

it failed

Step 9/21 : RUN go get -u -f github.com/anycable/anycable-go/cmd/anycable-go
---> Running in 4a95d2ef162e
runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed

runtime stack:
runtime.throw(0xa3b43e, 0xc)
/usr/local/go/src/runtime/panic.go:1112 +0x72
runtime.mlockGsignal(0xc00052c300)
/usr/local/go/src/runtime/os_linux_x86.go:72 +0x107
runtime.mpreinit(0xc000879880)
/usr/local/go/src/runtime/os_linux.go:341 +0x78
runtime.mcommoninit(0xc000879880)
/usr/local/go/src/runtime/proc.go:630 +0x108
runtime.allocm(0xc000041800, 0xa82448, 0x0)
/usr/local/go/src/runtime/proc.go:1390 +0x14e
runtime.newm(0xa82448, 0xc000041800)
/usr/local/go/src/runtime/proc.go:1704 +0x39
runtime.startm(0x0, 0xc0001b0401)
/usr/local/go/src/runtime/proc.go:1869 +0x12a
runtime.wakep(...)
/usr/local/go/src/runtime/proc.go:1953
runtime.resetspinning()
/usr/local/go/src/runtime/proc.go:2415 +0x93
runtime.schedule()
/usr/local/go/src/runtime/proc.go:2527 +0x2de
runtime.park_m(0xc000001080)
/usr/local/go/src/runtime/proc.go:2690 +0x9d
runtime.mcall(0x800000)
/usr/local/go/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc000517374)
/usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc000517374)
/usr/local/go/src/sync/waitgroup.go:130 +0x64
cmd/go/internal/work.(*Builder).Do(0xc000764b40, 0xc0000dda40)
/usr/local/go/src/cmd/go/internal/work/exec.go:187 +0x3ae
cmd/go/internal/work.InstallPackages(0xc0000b8040, 0x1, 0x1, 0xc000527508, 0x1, 0x1)
/usr/local/go/src/cmd/go/internal/work/build.go:605 +0xb2d
cmd/go/internal/get.runGet(0xebd5e0, 0xc0000b8040, 0x1, 0x1)
/usr/local/go/src/cmd/go/internal/get/get.go:185 +0x1fe
main.main()
/usr/local/go/src/cmd/go/main.go:189 +0x569

goroutine 1733 [IO wait]:
internal/poll.runtime_pollWait(0x7f8729cc7df8, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000965878, 0x72, 0x201, 0x200, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000965860, 0xc000246200, 0x200, 0x200, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc000527f40, 0xc000246200, 0x200, 0x200, 0xc00070a690, 0x447242, 0xc00070a6a0)
/usr/local/go/src/os/file.go:116 +0x71
bytes.(*Buffer).ReadFrom(0xc00098e7e0, 0xb2d5e0, 0xc000527f40, 0x7f8729cf9028, 0xc00098e7e0, 0xc00070a701)
/usr/local/go/src/bytes/buffer.go:204 +0xb1
io.copyBuffer(0xb2c9c0, 0xc00098e7e0, 0xb2d5e0, 0xc000527f40, 0x0, 0x0, 0x0, 0x4068a5, 0xc000964840, 0xc00070a7b0)
/usr/local/go/src/io/io.go:391 +0x2fc
io.Copy(...)
/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0xc000964840, 0xc00070a7b0)
/usr/local/go/src/os/exec/exec.go:310 +0x63
os/exec.(*Cmd).Start.func1(0xc00017b4a0, 0xc00038eba0)
/usr/local/go/src/os/exec/exec.go:436 +0x27
created by os/exec.(*Cmd).Start
/usr/local/go/src/os/exec/exec.go:435 +0x608

goroutine 1690 [IO wait]:
internal/poll.runtime_pollWait(0x7f872b6a8818, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000493bd8, 0x72, 0x201, 0x200, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000493bc0, 0xc000336200, 0x200, 0x200, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc00047e6e0, 0xc000336200, 0x200, 0x200, 0xc0006b6690, 0x447242, 0xc0006b66a0)
/usr/local/go/src/os/file.go:116 +0x71
bytes.(*Buffer).ReadFrom(0xc00098a930, 0xb2d5e0, 0xc00047e6e0, 0x7f8729cf9028, 0xc00098a930, 0xc0006b6701)
/usr/local/go/src/bytes/buffer.go:204 +0xb1
io.copyBuffer(0xb2c9c0, 0xc00098a930, 0xb2d5e0, 0xc00047e6e0, 0x0, 0x0, 0x0, 0x4068a5, 0xc000072660, 0xc0006b67b0)
/usr/local/go/src/io/io.go:391 +0x2fc
io.Copy(...)
/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0xc000072660, 0xc0006b67b0)
/usr/local/go/src/os/exec/exec.go:310 +0x63
os/exec.(*Cmd).Start.func1(0xc000a80b00, 0xc0001f8780)
/usr/local/go/src/os/exec/exec.go:436 +0x27
created by os/exec.(*Cmd).Start
/usr/local/go/src/os/exec/exec.go:435 +0x608

goroutine 1664 [IO wait]:
internal/poll.runtime_pollWait(0x7f872b6a89d8, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000965b78, 0x72, 0x201, 0x200, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000965b60, 0xc000156c00, 0x200, 0x200, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc000a12020, 0xc000156c00, 0x200, 0x200, 0xc0006b7e90, 0x447242, 0xc0006b7ea0)
/usr/local/go/src/os/file.go:116 +0x71
bytes.(*Buffer).ReadFrom(0xc00098e870, 0xb2d5e0, 0xc000a12020, 0x7f8729cf9028, 0xc00098e870, 0xc0006b7f01)
/usr/local/go/src/bytes/buffer.go:204 +0xb1
io.copyBuffer(0xb2c9c0, 0xc00098e870, 0xb2d5e0, 0xc000a12020, 0x0, 0x0, 0x0, 0x4068a5, 0xc00015eba0, 0xc0006b7fb0)
/usr/local/go/src/io/io.go:391 +0x2fc
io.Copy(...)
/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0xc00015eba0, 0xc0006b7fb0)
/usr/local/go/src/os/exec/exec.go:310 +0x63
os/exec.(*Cmd).Start.func1(0xc00017b600, 0xc00038ef40)
/usr/local/go/src/os/exec/exec.go:436 +0x27
created by os/exec.(*Cmd).Start
/usr/local/go/src/os/exec/exec.go:435 +0x608

goroutine 1626 [runnable]:
os.(*File).Read(0xc0005344b0, 0xc000b8e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/os/file.go:112 +0x240
io.copyBuffer(0x7f872835a568, 0xc000802280, 0xb2d5e0, 0xc0005344b0, 0xc000b8e000, 0x8000, 0x8000, 0x3, 0xc00038af30, 0x23)
/usr/local/go/src/io/io.go:405 +0x122
io.Copy(...)
/usr/local/go/src/io/io.go:364
cmd/go/internal/cache.(*Cache).put(0xc0004bc860, 0xcff4124e80c55fba, 0x600ad44b25c37c0a, 0xd9a767a1ebd24033, 0x131dde9687a6aafc, 0xb33940, 0xc0005344b0, 0x1, 0x0, 0x0, ...)
/usr/local/go/src/cmd/go/internal/cache/cache.go:420 +0x148
cmd/go/internal/cache.(*Cache).Put(...)
/usr/local/go/src/cmd/go/internal/cache/cache.go:403
cmd/go/internal/work.(*Builder).updateBuildID(0xc000764b40, 0xc000837040, 0xc00038af30, 0x23, 0x1, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/buildid.go:706 +0x530
cmd/go/internal/work.(*Builder).build(0xc000764b40, 0xc000837040, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:783 +0x1dbd
cmd/go/internal/work.(*Builder).Do.func2(0xc000837040)
/usr/local/go/src/cmd/go/internal/work/exec.go:118 +0x358
cmd/go/internal/work.(*Builder).Do.func3(0xc000517374, 0xc000764b40, 0xc0003def40)
/usr/local/go/src/cmd/go/internal/work/exec.go:178 +0x76
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:165 +0x38a

goroutine 1674 [IO wait]:
internal/poll.runtime_pollWait(0x7f872b6a8498, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000493458, 0x72, 0x201, 0x200, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000493440, 0xc000336000, 0x200, 0x200, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc00047e508, 0xc000336000, 0x200, 0x200, 0xc00071ce90, 0x447242, 0xc00071cea0)
/usr/local/go/src/os/file.go:116 +0x71
bytes.(*Buffer).ReadFrom(0xc00098a180, 0xb2d5e0, 0xc00047e508, 0x7f8729cf9028, 0xc00098a180, 0xc00071cf01)
/usr/local/go/src/bytes/buffer.go:204 +0xb1
io.copyBuffer(0xb2c9c0, 0xc00098a180, 0xb2d5e0, 0xc00047e508, 0x0, 0x0, 0x0, 0x4068a5, 0xc00015e5a0, 0xc00071cfb0)
/usr/local/go/src/io/io.go:391 +0x2fc
io.Copy(...)
/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0xc00015e5a0, 0xc00071cfb0)
/usr/local/go/src/os/exec/exec.go:310 +0x63
os/exec.(*Cmd).Start.func1(0xc000227b80, 0xc0001f8560)
/usr/local/go/src/os/exec/exec.go:436 +0x27
created by os/exec.(*Cmd).Start
/usr/local/go/src/os/exec/exec.go:435 +0x608

goroutine 1627 [runnable]:
syscall.Syscall6(0xf7, 0x1, 0x212, 0xc00014cf40, 0x1000004, 0x0, 0x0, 0x0, 0xc00014cf40, 0x0)
/usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
os.(*Process).blockUntilWaitable(0xc00042d440, 0x40db16, 0xc0008c1400, 0x1)
/usr/local/go/src/os/wait_waitid.go:31 +0x98
os.(*Process).wait(0xc00042d440, 0xa82120, 0xa82128, 0xa82118)
/usr/local/go/src/os/exec_unix.go:22 +0x39
os.(*Process).Wait(...)
/usr/local/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc0001e8c60, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:502 +0x60
os/exec.(*Cmd).Run(0xc0001e8c60, 0x28b995b37e, 0xec8260)
/usr/local/go/src/os/exec/exec.go:340 +0x5c
cmd/go/internal/work.(*Builder).runOut(0xc000764b40, 0xc000846280, 0xc0008b3310, 0x4a, 0x0, 0x0, 0x0, 0xc00016a780, 0xf, 0x14, ...)
/usr/local/go/src/cmd/go/internal/work/exec.go:1928 +0x5bc
cmd/go/internal/work.gcToolchain.gc(0xc000764b40, 0xc000846280, 0xc000417dd0, 0x23, 0xc0001f0380, 0xfa, 0x1b6, 0x0, 0x0, 0xc00014d500, ...)
/usr/local/go/src/cmd/go/internal/work/gc.go:142 +0xc98
cmd/go/internal/work.(*Builder).build(0xc000764b40, 0xc000846280, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:678 +0x1715
cmd/go/internal/work.(*Builder).Do.func2(0xc000846280)
/usr/local/go/src/cmd/go/internal/work/exec.go:118 +0x358
cmd/go/internal/work.(*Builder).Do.func3(0xc000517374, 0xc000764b40, 0xc0003def40)
/usr/local/go/src/cmd/go/internal/work/exec.go:178 +0x76
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:165 +0x38a

goroutine 1623 [syscall]:
syscall.Syscall6(0xf7, 0x1, 0x23b, 0xc000262f40, 0x1000004, 0x0, 0x0, 0x69, 0x6a, 0xc000262f80)
/usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
os.(*Process).blockUntilWaitable(0xc00038b650, 0x40db16, 0xc0007ca400, 0x1)
/usr/local/go/src/os/wait_waitid.go:31 +0x98
os.(*Process).wait(0xc00038b650, 0xa82120, 0xa82128, 0xa82118)
/usr/local/go/src/os/exec_unix.go:22 +0x39
os.(*Process).Wait(...)
/usr/local/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc00017b600, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:502 +0x60
os/exec.(*Cmd).Run(0xc00017b600, 0x28bfc4ba6f, 0xec8260)
/usr/local/go/src/os/exec/exec.go:340 +0x5c
cmd/go/internal/work.(*Builder).runOut(0xc000764b40, 0xc000846c80, 0xc000470d20, 0x49, 0x0, 0x0, 0x0, 0xc0007fc500, 0xf, 0x14, ...)
/usr/local/go/src/cmd/go/internal/work/exec.go:1928 +0x5bc
cmd/go/internal/work.gcToolchain.gc(0xc000764b40, 0xc000846c80, 0xc000759740, 0x23, 0xc0001880c0, 0x4c, 0xbc, 0x0, 0x0, 0xc000263500, ...)
/usr/local/go/src/cmd/go/internal/work/gc.go:142 +0xc98
cmd/go/internal/work.(*Builder).build(0xc000764b40, 0xc000846c80, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:678 +0x1715
cmd/go/internal/work.(*Builder).Do.func2(0xc000846c80)
/usr/local/go/src/cmd/go/internal/work/exec.go:118 +0x358
cmd/go/internal/work.(*Builder).Do.func3(0xc000517374, 0xc000764b40, 0xc0003def40)
/usr/local/go/src/cmd/go/internal/work/exec.go:178 +0x76
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:165 +0x38a

goroutine 1624 [syscall]:
syscall.Syscall6(0xf7, 0x1, 0x21a, 0xc0007d6f40, 0x1000004, 0x0, 0x0, 0x4d3b01, 0xc00062e000, 0xc0007d6f80)
/usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
os.(*Process).blockUntilWaitable(0xc00037c3c0, 0x40db16, 0xc0008e9c00, 0x1)
/usr/local/go/src/os/wait_waitid.go:31 +0x98
os.(*Process).wait(0xc00037c3c0, 0xa82120, 0xa82128, 0xa82118)
/usr/local/go/src/os/exec_unix.go:22 +0x39
os.(*Process).Wait(...)
/usr/local/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc000227b80, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:502 +0x60
os/exec.(*Cmd).Run(0xc000227b80, 0x28badc8de5, 0xec8260)
/usr/local/go/src/os/exec/exec.go:340 +0x5c
cmd/go/internal/work.(*Builder).runOut(0xc000764b40, 0xc0008463c0, 0xc0003523c0, 0x4d, 0x0, 0x0, 0x0, 0xc000847b80, 0x14, 0x14, ...)
/usr/local/go/src/cmd/go/internal/work/exec.go:1928 +0x5bc
cmd/go/internal/work.gcToolchain.gc(0xc000764b40, 0xc0008463c0, 0xc0002ebcb0, 0x23, 0xc0002381e0, 0x13e, 0x1ca, 0x0, 0x0, 0xc0007d7500, ...)
/usr/local/go/src/cmd/go/internal/work/gc.go:142 +0xc98
cmd/go/internal/work.(*Builder).build(0xc000764b40, 0xc0008463c0, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:678 +0x1715
cmd/go/internal/work.(*Builder).Do.func2(0xc0008463c0)
/usr/local/go/src/cmd/go/internal/work/exec.go:118 +0x358
cmd/go/internal/work.(*Builder).Do.func3(0xc000517374, 0xc000764b40, 0xc0003def40)
/usr/local/go/src/cmd/go/internal/work/exec.go:178 +0x76
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:165 +0x38a

goroutine 1621 [runnable]:
syscall.Syscall6(0xf7, 0x1, 0x1e3, 0xc000146f40, 0x1000004, 0x0, 0x0, 0x0, 0xc000146f40, 0x0)
/usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
os.(*Process).blockUntilWaitable(0xc000759950, 0x40db16, 0xc0008c1800, 0x1)
/usr/local/go/src/os/wait_waitid.go:31 +0x98
os.(*Process).wait(0xc000759950, 0xa82120, 0xa82128, 0xa82118)
/usr/local/go/src/os/exec_unix.go:22 +0x39
os.(*Process).Wait(...)
/usr/local/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc000227e40, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:502 +0x60
os/exec.(*Cmd).Run(0xc000227e40, 0x28b08f6a7c, 0xec8260)
/usr/local/go/src/os/exec/exec.go:340 +0x5c
cmd/go/internal/work.(*Builder).runOut(0xc000764b40, 0xc00083e280, 0xc0002c0780, 0x5e, 0x0, 0x0, 0x0, 0xc0009d0f00, 0x15, 0x28, ...)
/usr/local/go/src/cmd/go/internal/work/exec.go:1928 +0x5bc
cmd/go/internal/work.gcToolchain.gc(0xc000764b40, 0xc00083e280, 0xc00038ab40, 0x23, 0xc000242000, 0x347, 0x57e, 0x0, 0x0, 0xc000147500, ...)
/usr/local/go/src/cmd/go/internal/work/gc.go:142 +0xc98
cmd/go/internal/work.(*Builder).build(0xc000764b40, 0xc00083e280, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:678 +0x1715
cmd/go/internal/work.(*Builder).Do.func2(0xc00083e280)
/usr/local/go/src/cmd/go/internal/work/exec.go:118 +0x358
cmd/go/internal/work.(*Builder).Do.func3(0xc000517374, 0xc000764b40, 0xc0003def40)
/usr/local/go/src/cmd/go/internal/work/exec.go:178 +0x76
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:165 +0x38a

goroutine 1622 [syscall]:
syscall.Syscall6(0xf7, 0x1, 0x22a, 0xc0003b2f40, 0x1000004, 0x0, 0x0, 0x4d3b01, 0xc000965920, 0xc0003b2f80)
/usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
os.(*Process).blockUntilWaitable(0xc0007596e0, 0x40db16, 0xc0000df400, 0x1)
/usr/local/go/src/os/wait_waitid.go:31 +0x98
os.(*Process).wait(0xc0007596e0, 0xa82120, 0xa82128, 0xa82118)
/usr/local/go/src/os/exec_unix.go:22 +0x39
os.(*Process).Wait(...)
/usr/local/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc00017b4a0, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:502 +0x60
os/exec.(*Cmd).Run(0xc00017b4a0, 0x28bc016458, 0xec8260)
/usr/local/go/src/os/exec/exec.go:340 +0x5c
cmd/go/internal/work.(*Builder).runOut(0xc000764b40, 0xc000846b40, 0xc000a15b80, 0x4b, 0x0, 0x0, 0x0, 0xc0007fc280, 0x10, 0x14, ...)
/usr/local/go/src/cmd/go/internal/work/exec.go:1928 +0x5bc
cmd/go/internal/work.gcToolchain.gc(0xc000764b40, 0xc000846b40, 0xc000758930, 0x23, 0xc00023a280, 0x19e, 0x259, 0x0, 0x0, 0xc0003b3500, ...)
/usr/local/go/src/cmd/go/internal/work/gc.go:142 +0xc98
cmd/go/internal/work.(*Builder).build(0xc000764b40, 0xc000846b40, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:678 +0x1715
cmd/go/internal/work.(*Builder).Do.func2(0xc000846b40)
/usr/local/go/src/cmd/go/internal/work/exec.go:118 +0x358
cmd/go/internal/work.(*Builder).Do.func3(0xc000517374, 0xc000764b40, 0xc0003def40)
/usr/local/go/src/cmd/go/internal/work/exec.go:178 +0x76
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:165 +0x38a

goroutine 1625 [syscall]:
syscall.Syscall6(0xf7, 0x1, 0x184, 0xc000148f40, 0x1000004, 0x0, 0x0, 0x4d3b01, 0xc00015f140, 0xc000148f80)
/usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
os.(*Process).blockUntilWaitable(0xc000922d80, 0x40db16, 0xc0000df800, 0x1)
/usr/local/go/src/os/wait_waitid.go:31 +0x98
os.(*Process).wait(0xc000922d80, 0xa82120, 0xa82128, 0xa82118)
/usr/local/go/src/os/exec_unix.go:22 +0x39
os.(*Process).Wait(...)
/usr/local/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc00017af20, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:502 +0x60
os/exec.(*Cmd).Run(0xc00017af20, 0x28a1e0d6c4, 0xec8260)
/usr/local/go/src/os/exec/exec.go:340 +0x5c
cmd/go/internal/work.(*Builder).runOut(0xc000764b40, 0xc000836c80, 0xc0003dc5f0, 0x44, 0x0, 0x0, 0x0, 0xc000499400, 0x32, 0x50, ...)
/usr/local/go/src/cmd/go/internal/work/exec.go:1928 +0x5bc
cmd/go/internal/work.gcToolchain.gc(0xc000764b40, 0xc000836c80, 0xc00038bb00, 0x23, 0xc0001da000, 0x327, 0x4d0, 0xc00038b8c0, 0x23, 0xc000149501, ...)
/usr/local/go/src/cmd/go/internal/work/gc.go:142 +0xc98
cmd/go/internal/work.(*Builder).build(0xc000764b40, 0xc000836c80, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:678 +0x1715
cmd/go/internal/work.(*Builder).Do.func2(0xc000836c80)
/usr/local/go/src/cmd/go/internal/work/exec.go:118 +0x358
cmd/go/internal/work.(*Builder).Do.func3(0xc000517374, 0xc000764b40, 0xc0003def40)
/usr/local/go/src/cmd/go/internal/work/exec.go:178 +0x76
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:165 +0x38a

goroutine 1628 [syscall]:
syscall.Syscall6(0xf7, 0x1, 0x21e, 0xc0007d2f40, 0x1000004, 0x0, 0x0, 0x4d3b01, 0xc000493c80, 0xc0007d2f80)
/usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
os.(*Process).blockUntilWaitable(0xc000923440, 0x40db16, 0xc0008c4400, 0x1)
/usr/local/go/src/os/wait_waitid.go:31 +0x98
os.(*Process).wait(0xc000923440, 0xa82120, 0xa82128, 0xa82118)
/usr/local/go/src/os/exec_unix.go:22 +0x39
os.(*Process).Wait(...)
/usr/local/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc000a80b00, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:502 +0x60
os/exec.(*Cmd).Run(0xc000a80b00, 0x28bb4e3f57, 0xec8260)
/usr/local/go/src/os/exec/exec.go:340 +0x5c
cmd/go/internal/work.(*Builder).runOut(0xc000764b40, 0xc000846640, 0xc0007c02d0, 0x4b, 0x0, 0x0, 0x0, 0xc000847cc0, 0x12, 0x14, ...)
/usr/local/go/src/cmd/go/internal/work/exec.go:1928 +0x5bc
cmd/go/internal/work.gcToolchain.gc(0xc000764b40, 0xc000846640, 0xc0009230b0, 0x23, 0xc000194000, 0x11e, 0x1a2, 0x0, 0x0, 0xc0007d3500, ...)
/usr/local/go/src/cmd/go/internal/work/gc.go:142 +0xc98
cmd/go/internal/work.(*Builder).build(0xc000764b40, 0xc000846640, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:678 +0x1715
cmd/go/internal/work.(*Builder).Do.func2(0xc000846640)
/usr/local/go/src/cmd/go/internal/work/exec.go:118 +0x358
cmd/go/internal/work.(*Builder).Do.func3(0xc000517374, 0xc000764b40, 0xc0003def40)
/usr/local/go/src/cmd/go/internal/work/exec.go:178 +0x76
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:165 +0x38a

goroutine 1719 [IO wait]:
internal/poll.runtime_pollWait(0x7f8729cc8178, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc0005d26d8, 0x72, 0x201, 0x200, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0005d26c0, 0xc00014e400, 0x200, 0x200, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc00054e050, 0xc00014e400, 0x200, 0x200, 0xc00071ee90, 0xc00003c800, 0xc00071eea0)
/usr/local/go/src/os/file.go:116 +0x71
bytes.(*Buffer).ReadFrom(0xc00098e360, 0xb2d5e0, 0xc00054e050, 0x7f8729cf9028, 0xc00098e360, 0xc00071ef01)
/usr/local/go/src/bytes/buffer.go:204 +0xb1
io.copyBuffer(0xb2c9c0, 0xc00098e360, 0xb2d5e0, 0xc00054e050, 0x0, 0x0, 0x0, 0x4068a5, 0xc000492000, 0xc00071efb0)
/usr/local/go/src/io/io.go:391 +0x2fc
io.Copy(...)
/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0xc000492000, 0xc00071efb0)
/usr/local/go/src/os/exec/exec.go:310 +0x63
os/exec.(*Cmd).Start.func1(0xc00017af20, 0xc00038e480)
/usr/local/go/src/os/exec/exec.go:436 +0x27
created by os/exec.(*Cmd).Start
/usr/local/go/src/os/exec/exec.go:435 +0x608

goroutine 1662 [IO wait]:
internal/poll.runtime_pollWait(0x7f8729cc7d18, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc0004938d8, 0x72, 0x201, 0x200, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0004938c0, 0xc000156000, 0x200, 0x200, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc00047e070, 0xc000156000, 0x200, 0x200, 0xc0006bae90, 0x447242, 0xc0006baea0)
/usr/local/go/src/os/file.go:116 +0x71
bytes.(*Buffer).ReadFrom(0xc000b08fc0, 0xb2d5e0, 0xc00047e070, 0x7f8729cf9028, 0xc000b08fc0, 0xc0006baf01)
/usr/local/go/src/bytes/buffer.go:204 +0xb1
io.copyBuffer(0xb2c9c0, 0xc000b08fc0, 0xb2d5e0, 0xc00047e070, 0x0, 0x0, 0x0, 0x4068a5, 0xc000072a20, 0xc0006bafb0)
/usr/local/go/src/io/io.go:391 +0x2fc
io.Copy(...)
/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0xc000072a20, 0xc0006bafb0)
/usr/local/go/src/os/exec/exec.go:310 +0x63
os/exec.(*Cmd).Start.func1(0xc000227e40, 0xc0004bcca0)
/usr/local/go/src/os/exec/exec.go:436 +0x27
created by os/exec.(*Cmd).Start
/usr/local/go/src/os/exec/exec.go:435 +0x608

goroutine 1705 [IO wait]:
internal/poll.runtime_pollWait(0x7f872b6a8ab8, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00015f1b8, 0x72, 0x201, 0x200, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00015f1a0, 0xc000246000, 0x200, 0x200, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc0002368f0, 0xc000246000, 0x200, 0x200, 0xc0001b2e90, 0x447242, 0xc0001b2ea0)
/usr/local/go/src/os/file.go:116 +0x71
bytes.(*Buffer).ReadFrom(0xc000b099e0, 0xb2d5e0, 0xc0002368f0, 0x7f8729cf9028, 0xc000b099e0, 0xc0001b2f01)
/usr/local/go/src/bytes/buffer.go:204 +0xb1
io.copyBuffer(0xb2c9c0, 0xc000b099e0, 0xb2d5e0, 0xc0002368f0, 0x0, 0x0, 0x0, 0x4068a5, 0xc000492960, 0xc0001b2fb0)
/usr/local/go/src/io/io.go:391 +0x2fc
io.Copy(...)
/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0xc000492960, 0xc0001b2fb0)
/usr/local/go/src/os/exec/exec.go:310 +0x63
os/exec.(*Cmd).Start.func1(0xc0001e8c60, 0xc0004288a0)
/usr/local/go/src/os/exec/exec.go:436 +0x27
created by os/exec.(*Cmd).Start
/usr/local/go/src/os/exec/exec.go:435 +0x608
ERROR: Service 'web' failed to build: The command '/bin/sh -c go get -u -f github.com/anycable/anycable-go/cmd/anycable-go' returned a non-zero code: 2

The long and winding roadmap

It's time to make the codebase well-orgnazed, add tests and enhance functionality!

• Restructure it!
  • Migrate to dep
  • Organize code in packages
  • More tests!
• Improve it!
  • Improve configuration ( namespaced env vars)
  • Migrate to logrus apex/log; improve logging (see #24)
• Make it rock!
  • Provide metrics (TBD)
  • Add unique identifiers to connections and use them as log tags (NanoID https://github.com/matoous/go-nanoid ?)
  • Abstractize pub/sub listener
  • Provide HTTP publish endpoint
  • File-based configuration (anycable.toml?)`
  • Add health checking (see anycable/anycable#29)
• It's magic!
  • Consider providing plugin support (e.g. pub/sub adapters, logging hooks)
  • Embed mruby to move some work from RPC to Go

Nil pointer dereference

Hi, I'm trying anycable with my Rails app for the first time. I've just set everything up and I'm getting the following error:

$ anycable-go -rpc=0.0.0.0:50051 -redis=redis://localhost:6379/5 -redis_channel=anycable -addr=0.0.0.0:3334
Running websocket server on 0.0.0.0:3334 at /cable
RPC Error: rpc error: code = 2 desc = no reason given
2017/01/29 21:22:12 http: panic serving [::1]:57629: runtime error: invalid memory address or nil pointer dereference
goroutine 97 [running]:
net/http.(*conn).serve.func1(0xc82022a080)
	/usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:1389 +0xc1
panic(0x508140, 0xc820010110)
	/usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:443 +0x4e9
main.serveWs(0xbf8220, 0xc8202361a0, 0xc82025c0e0)
	/Users/palkan/lib/go/src/github.com/anycable/anycable-go/server.go:141 +0x254
net/http.HandlerFunc.ServeHTTP(0x6a78f8, 0xbf8220, 0xc8202361a0, 0xc82025c0e0)
	/usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc820076c00, 0xbf8220, 0xc8202361a0, 0xc82025c0e0)
	/usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc82009ee00, 0xbf8220, 0xc8202361a0, 0xc82025c0e0)
	/usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc82022a080)
	/usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
	/usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2137 +0x44e

I don't know what I'm doing wrong. Can somebody give me some tips?

Consider using alpine:latest instead of edge

Hi,

Following the warning on Alpine Linux wiki page:

"edge" is under constant development so be careful using it in production. It is possible that bugs in "edge" could cause data loss or could break your system.

I think it would be a great idea to use the alpine:latest version (or a fixed version of alpine), as the edge version is not suitable for production environment.

Exchanging the ActionCable-JSON "frontend" with other Protocol?

I'm currently working on using a custom (ProtoBuf) wrapper for the communication between client and AnyCable server. So I want to get rid of the JSON load ActionCable uses and replace it with specific ProtoBuf definitions for my API (ActionChannel) calls.
The main reasons for this are:

1. Massive savings in Bandwidth
2. Implicitly documented API calls (through .proto files)
3. No datatype-problems (like using int64 in JSON which may or may not be supported)

I figured it might be easiest to leave AnyCable's gRPC-communication with Rails as it is (because it's quite optimal) and add a conversion-layer to AnyCable to convert Rails' ActionCable-responses (over gRPC) into specific ProtoBuf-loads. Unfortunately I've to get a grip on Go first.

My questions (or rather my pleas) to you as AnyCable gurus is: where would you recommend implementing such behavior in the AnyCable code? And what do you think about my approach on ProtoBuf conversion?

Thank you very much in advance! <3

HSTS support

hi guys,

We've really tried googling this answer to no avail. Either our googling skills suck... or, actually probably that :-)

We're using Anycable-go and we want to make sure our socket connections implement HSTS (we recently had a penetration test done and that came up as a minor issue).

How do we do this when using anycable go?

thanks!

CORS config

We are using anycable - go as an WS server for our rails app in api only mode.
Is it possible to configure CORS in this case? Or it should be done maybe at nginx level?

Thanks.

Mass broadcast

Может сделать возможность массовой рассылки на все открытые соединения?

Может быть удобно при деплое, дать всем знать, что сейчас планово оборвется связь и нужно без паники переконнектиться и быть готовым обновиться.

В целом изменение в коде будет небольшим.

Allow same origin as Host configuration is always failing because HTTP_HOST is always blank

AnyCable-Go version:
1.0.0 preview

AnyCable gem version:
1.0.0 preview

What did you do?

I have anycable-go server running behind a Nginx reverse proxy. From Nginx reverse proxy, I set the headers for Host. I ran the go server listening to 8080 port and have configured to fetch headers host, cookie and origin. My aim is to pass both host and origin as part of the header so that allow_same_origin_as_host configuration should succeed in allow_request_origin? (Forgery protection check) method.

Ngnix configuration

location /cable {
proxy_pass http://localhost:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_set_header Http_AnyCable_Host $http_host;
}

Command to start go server
anycable-go --headers=cookie,origin,host,http_anycable_host --redis_url=redis://localhost:6379/4 --redis_channel=anycable-development --host=localhost --port=8080 --log_level=debug

Anycable go server log

D 2020-04-17T21:12:46.053Z context=node sid=9Pm42H7tho_lyeZDEBhnHd Disconnect {"ltags":["Member ID: XX"],"member":"Z2lkOi8vXXb2XX51cy1tZW50b3ItYmFzZS9NZW1iZXIvNTA"} /cable map[REMOTE_ADDR:127.0.0.1 cookie: current_locale=en;client_time_difference=-836 host: http_anycable_host:abc.something.com origin:https://abc.something.com] [{"channel":"ChatChannel","context_id":XXXXX}]

What did you expect to happen?

But the allow_same_origin_as_host check failed because HTTP_HOST value is always blank. If I try to send the $http_host value in some other custom header say "http_anycable_host", then I could see its (HTTP_ANYCABLE_HOST) value in the request headers.

In any cable go server log, I couldn't find the value for the host header.

Could you please help me understand why the HTTP_HOST value is always blank?
@palkan @Envek @sponomarev

Totally different environments seem to send messages to the same one?

Hi, I am using AnyCable in Rails and have installed AnyCable Go in Kuberentes with the helm chart.
I have three totally different environments for my app (dev/staging/prod) so I have three different AnyCable-Go deployments, each pointing to the relevant environment of the Rails app for the RPC service. I think that everything is configured correctly in that the services etc have all different namespaces and names and they are pointing to the right things for each environment. However, if I have three tabs for my app in my browser, one for each environment, and with the Rails console I broadcast a message to one environment, ALL the three environments receive the same message, even though they are connecting to different domains each of which is connecting to a different RPC host.

The only thing in common is the single Redis instance, but I am specifying different database numbers in the Redis URLs. Can this explain the problem? Can multiple AnyCable instances share the same Redis instances with different databases?

What could be causing a message sent to one instance of AnyCable being received by the others as well?

I'm stuck, so thanks a lot in advance for any help! :)

Permessage deflate?

Disclaimer: I'm just getting into WebSocket and stuff.

I'm just experimenting on replacing our project's current "HTTP via Rails plus WebSocket via JS/Socket.io"-APIs with ActionCable/WebSockets completely. Performance is quite important, therefore I came across AnyCable quite early. As we're often sending data packets of several kilobytes to the client via HTTP, we had automatic gzip/deflate in place through a proxy.

But what if we wanted to send these big responses over WebSocket? I found there's a, IETF proposal "permessage deflate" which handles compression if the clients support it (which I can ensure, we're building the client ourselves). As far as I understood, it's a feature on the WebSocket level - something anycable-go handles.
As I couldn't find "deflate" in this repo's source: are there any plans on supporting compression?

Make disconnector graceful shutdown timeout configurable

Having hardcoded timeout is too opinionated (5s doesn't fit everyone):

anycable-go/node/disconnect_queue.go

Lines 13 to 14 in 775acdf

	// TODO: make configurable
	waitTime = 5 * time.Second

As as temporary solution, we need to make this timeout configurable.
In the future, we need to think about a better way to make disconnection stable (since application code could rely on disconnect and unsubscribed callbacks).

DOD

• Add CLI option (--disconnect_graceful_period?)
• Document this behavior and potential caveats

Many dead connections on the server

Hello, recently we have deployed anycable to the production server and saw that connection count constantly increases, from 500 to 2k connections by night. After investigation we found out that we have dead clients connections (10%-15%). If client closes his laptop or disconnects from internet, connection doesn't close for a long time with send-q in netstat (go send ping and doesn't receive err). We tried to enable TCP keepalive but it didn't work. I guess we need to implement something like pong from client and close the connection when pong didn't receive. Maybe you have experience with this problem of detect dead client.

client message size as option?

Hi, I'm using anycable-go 0.6.0-preview2. Client sends base64-encoded image to server and its disconnected. How to increase client message size limit?

I forked repo, comment this line and install from fork.

But nothing has changed. Thanks

ping + log

Надо иметь возможность отделить вывод пингов в лог от остальных броадкастов. Через параметры запуска или перенести пинги из DEBUG в INFO ( но это не бьется с текущим броадкастом, который логгирует все сообщения выводит в дебаг лог ).

Потому что логгирование, особенно живое в девелопменте, практически смысла не имеет, все погребается пингами в клочья.

я себе локально сделал broadcast_silent для дева, но это надо как-то вытащить на какой-то разумный управляемый уровень.

Я не умею в go поэтому PR не могу сделать адекватный.

Anycable.broadcast не работает!

AnyCable-Go version: 0.6.0.preview5

AnyCable gem version: anycable-rails: 0.5.3

Не работает Anycable.broadcast.

Anycable.broadcast( stream_id_for(recipient), {test: { broadcast: :failed! } } )

эникейбл-го выдает ошибку:

context=node Failed to parse pubsub message '{"stream":"msgs:2","data":{"test":{"broadcast":"failed!"}}}' with error: json: cannot unmarshal object into Go struct field StreamMessage.data of type string

Трансмит работает нормально и клиент получает сообщение нормально распарсированное.

Если попытаться отправить строку ошибки нет, но опять же ничего не приходит.

The "--metrics_host" option does not work in v1.0.0

AnyCable-Go version:
1.0.0-33085f2

AnyCable gem version:
0.6.5

What did you do?

I'm trying to specify the explicit "--metrics_host" option for Prometheus.
$ /opt/anycable-go/v1.0.0/anycable-go-linux-amd64 --host 10.64.0.41 --log_level debug --metrics_host 10.64.0.41 --metrics_http /metrics --metrics_port 9109 --port 3334 --rpc_concurrency 3 --redis_url redis://10.64.0.12:6379/10

What did you expect to happen?

When I do the same for anycable-go 0.6.4 I can receive Prometheus metrics inside my local network:

$ curl -v http://10.64.0.41:9109/metrics
*   Trying 10.64.0.41...
* Connected to 10.64.0.41 (10.64.0.41) port 9109 (#0)
> GET /metrics HTTP/1.1
> Host: 10.64.0.41:9109
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Fri, 03 Jul 2020 13:05:45 GMT
< Content-Length: 1973
< Content-Type: text/plain; charset=utf-8
< 

# HELP anycable_go_failed_broadcast_msg_total The total number of unrecognized messages received through PubSub
# TYPE anycable_go_failed_broadcast_msg_total counter
anycable_go_failed_broadcast_msg_total 0

# HELP anycable_go_rpc_call_total The total number of RPC calls
# TYPE anycable_go_rpc_call_total counter
anycable_go_rpc_call_total 134
...

What actually happened?

Any other network settings were not changed (only anycable-go was upgraded from 0.6.4 to 1.0.0), but the connection is refused

$ curl -v http://10.64.0.41:9109/metrics
*   Trying 10.64.0.41...
* connect to 10.64.0.41 port 9109 failed: Connection refused
* Failed to connect to 10.64.0.41 port 9109: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 10.64.0.41 port 9109: Connection refused

At the same time log shows that the metrics server was started at http://localhost:9109/metrics instead of 10.64.0.41

Jul 03 06:42:03 d38e3de2910c systemd[1]: Started AnyCable WebSocket Server.
Jul 03 06:42:03 d38e3de2910c anycable-go-linux-amd64[20979]: I 2020-07-03T13:42:03.098Z context=main Starting AnyCable 1.0.0-33085f2 (pid: 20979, open file limit: 1048576)
Jul 03 06:42:03 d38e3de2910c anycable-go-linux-amd64[20979]: D 2020-07-03T13:42:03.098Z context=disconnector Calls rate: 10ms
Jul 03 06:42:03 d38e3de2910c anycable-go-linux-amd64[20979]: I 2020-07-03T13:42:03.098Z context=main Handle WebSocket connections at http://10.64.0.41:3334/cable
Jul 03 06:42:03 d38e3de2910c anycable-go-linux-amd64[20979]: I 2020-07-03T13:42:03.098Z context=main Handle health connections at http://10.64.0.41:3334/health
Jul 03 06:42:03 d38e3de2910c anycable-go-linux-amd64[20979]: D 2020-07-03T13:42:03.098Z context=http Starting WebSocket server at http://10.64.0.41:3334
Jul 03 06:42:03 d38e3de2910c anycable-go-linux-amd64[20979]: I 2020-07-03T13:42:03.099Z context=rpc RPC controller initialized: localhost:50051 (concurrency: 3, proto_versions: v0,v1)
Jul 03 06:42:03 d38e3de2910c anycable-go-linux-amd64[20979]: I 2020-07-03T13:42:03.099Z context=pubsub Subscribed to Redis channel: __anycable__
Jul 03 06:42:03 d38e3de2910c anycable-go-linux-amd64[20979]: I 2020-07-03T13:42:03.099Z context=metrics Serve metrics at http://localhost:9109/metrics
Jul 03 06:42:03 d38e3de2910c anycable-go-linux-amd64[20979]: D 2020-07-03T13:42:03.099Z context=http Starting Metrics server at http://localhost:9109

Redis Sentinel Support

Does anycable-go support Redis Sentinel?

Websocket connections not closed using Kong load balancer

Tell us about your environment

AnyCable-Go version:
v0.6.3
AnyCable gem version:
v0.6.3

What did you do?

Use kong as a load balancer.

What did you expect to happen?

Websocket connections are closed after some time.

What actually happened?

Websocket connections add up very fast until kong reaches its limit (8-10k) and restarts with

[alert] 33#0: 16384 worker_connections are not enough

Before kong we used the native kubernetes nginx as a load-balancer and the connections got closed properly. We usually had around 1k connections at peak times.
First we thougt the problem was that sticky sessions were not working properly. This problem should be fixed but we still see the same issue. (first bump in the picture is before we used sticky sessions hash_on: ip)

EDIT: a graph from before we were using kong:

Are there any special settings that need to be set in load-balancer?
Any other idea how to resolve that problem?

Thank You

Consider updating base image from `scratch` to `alpine`

scratch does not contain /bin/sh, or anything else. I can't run it with docker run -it --entrypoint=/bin/sh anycable/anycable-go:v0.6.2 -c "--redis-url=redis://localhost:6379/5 --http-health-port=80 --rpc-host=0.0.0.0:50001" and use docker exec -it anycable /bin/sh to get ssh into the running container.

Env Support for Anycable-go

I was wondering if we could get a version of anycable which supports env variables? (https://github.com/iij/mruby-env). I would make a branch and do a pull request myself, but I don't have the proper permissions to do so. To make such a version, all one would need to do is add the line conf.gem :github => 'iij/mruby-env' to the build_config.rb file. My organization would benefit tremendously from our anycable-go having access to env variables, so if this is at all possible it would be greatly appreciated!

-Dustin McRorie

Connection to the gRPC server gets stuck

AnyCable-Go version: 0.6.3
AnyCable gem version: 0.6.3 (same anycable-rails version)
gRPC gem version: 1.20.0
nginx version: 1.17.3

What did you do?

1. Use nginx + grpc module setup. Gist link
2. Run two instance of gRPC servers via bundle exec anycable --rpc_host 0.0.0.0:50052 and bundle exec anycable --rpc_host 0.0.0.0:50051
3. Run an instance of anycable-go via anycable-go --headers=origin,cookie --debug=true --rpc_host=localhost:50050
4. Subscribe to a channel. Nothing fancy here, using the JS ActionCable.subscribe.
5. Perform actions on the subscription periodically every 10 seconds in JS – subscription.perform 'do_stuff'.
6. Stop both gRPC anycable instances without de-registering any from nginx.
7. On the next do_stuff action, the anycable-go server receives error 502 from nginx since both gRPC servers are gone.

What did you expect to happen?

The anycable-go server to raise an error similar to when no connection to the gRPC is available (Perform error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure,) and retry communicating with the gRPC server on the next do_stuff action.

What actually happened?

After 7., no attempt to send requests to the gRPC server are made (nothing is logged in the anycable-go server and nothing is available in the nginx access log), even if the gRPC servers are started up again. Meanwhile, the client get successful ping messages and can receive broadcasts and through the WS.

If another client gets to subscribe to the same channel they'll either 1) get an error forcing them to reconnect when the gRPC servers are all down or 2) successfully subscribe and perform actions when the gRPC servers are up. The first client will still remain "stuck" however.

Bottom line is that performing actions on a subscription after getting error 502 blocks all new actions from being performed by the anycable-go server for a particular client/subscription.

Could you please give some directions on how to deal with this scenario? One possibility is to 'ack' for actions on the client side and reconnect altogether, but it adds some complexity.

How to install this for mac?

I am looking at these files for anycable and my mac will not let me install them because they are not actual files like .dmg, I do not know how to use go but I saw that as the other option for installing anycable-go. How do we install this?

Cannot install latest stable release

AnyCable-Go version: 0.5.4

What did you do?

Ran go get -u -f github.com/anycable/anycable-go/cmd/anycable-go

What did you expect to happen?

Should successfully install anycable-go.

What actually happened?

The following error:

# github.com/anycable/anycable-go/vendor/github.com/mitchellh/go-mruby
In file included from ../.gvm/pkgsets/go1.9.7/global/src/github.com/anycable/anycable-go/vendor/github.com/mitchellh/go-mruby/args.go:5:0:
./gomruby.h:12:19: fatal error: mruby.h: No such file or directory
 #include <mruby.h>
                   ^
compilation terminated.

Using a pre-compiled binary still works.

Homebrew formula

Let's make it super-simple to install anycable-go on MacOS!

Time for Notifier Disconnected

Do you know what is the time about notifier disconnection ?