Ion nodes die without logs

[ionorg]

Ion nodes die without logs

[adwpc]

sfu memory increase fixed!

[adwpc]

Maybe OOM?

[oleh-ozimok]

I constantly get an error:

runtime stack:
runtime.throw(0xf35532, 0x16)
	/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sysMap(0xc014000000, 0x4000000, 0x1766df8)
	/usr/local/go/src/runtime/mem_linux.go:169 +0xc5
runtime.(*mheap).sysAlloc(0x174e480, 0x2000, 0x2000, 0x610)
	/usr/local/go/src/runtime/malloc.go:701 +0x1cd
runtime.(*mheap).grow(0x174e480, 0x1, 0xffffffff)
	/usr/local/go/src/runtime/mheap.go:1255 +0xa3
runtime.(*mheap).allocSpanLocked(0x174e480, 0x1, 0x1766e08, 0x7f023c105d00)
	/usr/local/go/src/runtime/mheap.go:1170 +0x266
runtime.(*mheap).alloc_m(0x174e480, 0x1, 0x43, 0x7f023c105d00)
	/usr/local/go/src/runtime/mheap.go:1022 +0xc2
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:1093 +0x4c
runtime.systemstack(0xc0002a3840)
	/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1146

goroutine 321914 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0011d1a38 sp=0xc0011d1a30 pc=0x45c670
runtime.(*mheap).alloc(0x174e480, 0x1, 0x10043, 0x0)
	/usr/local/go/src/runtime/mheap.go:1092 +0x8a fp=0xc0011d1a88 sp=0xc0011d1a38 pc=0x425e9a
runtime.(*mcentral).grow(0x174f8f8, 0x0)
	/usr/local/go/src/runtime/mcentral.go:255 +0x7b fp=0xc0011d1ac8 sp=0xc0011d1a88 pc=0x417dfb
runtime.(*mcentral).cacheSpan(0x174f8f8, 0x76a07f234310fbfb)
	/usr/local/go/src/runtime/mcentral.go:106 +0x2fe fp=0xc0011d1b28 sp=0xc0011d1ac8 pc=0x41791e
runtime.(*mcache).refill(0x7f023f27e008, 0x43)
	/usr/local/go/src/runtime/mcache.go:138 +0x85 fp=0xc0011d1b48 sp=0xc0011d1b28 pc=0x4173c5
runtime.(*mcache).nextFree(0x7f023f27e008, 0xa0020043, 0x0, 0x80, 0xc013f4ec00)
	/usr/local/go/src/runtime/malloc.go:854 +0x87 fp=0xc0011d1b80 sp=0xc0011d1b48 pc=0x40bcf7
runtime.mallocgc(0x500, 0xd91580, 0x1, 0xc0011d1c68)
	/usr/local/go/src/runtime/malloc.go:1022 +0x793 fp=0xc0011d1c20 sp=0xc0011d1b80 pc=0x40c633
runtime.makeslice(0xd91580, 0x4b8, 0x4b8, 0xc0011d1d90)
	/usr/local/go/src/runtime/slice.go:49 +0x6c fp=0xc0011d1c50 sp=0xc0011d1c20 pc=0x44549c
github.com/pion/srtp.growBufferSize(...)
	/go/pkg/mod/github.com/pion/[email protected]/util.go:11
github.com/pion/srtp.(*Context).encryptRTP(0xc000fa57c0, 0x0, 0x0, 0x0, 0xc013ddf4d0, 0xc013f7600c, 0x4a2, 0x5a8, 0xc013ffd900, 0xc0006f2900, ...)
	/go/pkg/mod/github.com/pion/[email protected]/srtp.go:87 +0x406 fp=0xc0011d1ce0 sp=0xc0011d1c50 pc=0xbe4786
github.com/pion/srtp.(*SessionSRTP).writeRTP(0xc000a86d20, 0xc013ddf4d0, 0xc013f7600c, 0x4a2, 0x5a8, 0x0, 0x0, 0x0)
	/go/pkg/mod/github.com/pion/[email protected]/session_srtp.go:130 +0x128 fp=0xc0011d1da0 sp=0xc0011d1ce0 pc=0xbe24e8
github.com/pion/srtp.(*WriteStreamSRTP).WriteRTP(...)
	/go/pkg/mod/github.com/pion/[email protected]/stream_srtp.go:124
github.com/pion/webrtc/v2.(*RTPSender).sendRTP(0xc0002f1540, 0xc013ddf4d0, 0xc013f7600c, 0x4a2, 0x5a8, 0x89e98b, 0x0, 0xf3ed3a)
	/go/pkg/mod/github.com/pion/webrtc/[email protected]/rtpsender.go:158 +0x158 fp=0xc0011d1e48 sp=0xc0011d1da0 pc=0xc646b8
github.com/pion/webrtc/v2.(*Track).WriteRTP(0xc0006330e0, 0xc013ddf4d0, 0xc0011d1ef8, 0x1)
	/go/pkg/mod/github.com/pion/webrtc/[email protected]/track.go:160 +0x141 fp=0xc0011d1eb8 sp=0xc0011d1e48 pc=0xc6ebc1
github.com/pion/ion/pkg/rtc/transport.(*WebRTCTransport).WriteRTP(0xc0002bc5a0, 0xc013ddf4d0, 0x0, 0x0)
	/go/src/github.com/pion/ion/pkg/rtc/transport/webrtctransport.go:366 +0x103 fp=0xc0011d1f18 sp=0xc0011d1eb8 pc=0xcc21e3
github.com/pion/ion/pkg/rtc.(*Router).out.func1.1(0xc000136e70, 0xc013ddf4d0)
	/go/src/github.com/pion/ion/pkg/rtc/router.go:148 +0xd5 fp=0xc0011d1fd0 sp=0xc0011d1f18 pc=0xcc9195
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0011d1fd8 sp=0xc0011d1fd0 pc=0x45e5c1
created by github.com/pion/ion/pkg/rtc.(*Router).out.func1
	/go/src/github.com/pion/ion/pkg/rtc/router.go:140 +0x13d```

[adwpc]

Can any one tell me which go version

[adwpc]

I keep publishing one stream for 4h, the mem is stable 236mb

ps -v                                                                                                     20:52:03
  PID STAT      TIME  SL  RE PAGEIN      VSZ    RSS   LIM     TSIZ  %CPU %MEM COMMAND
14062 S     11:18.65   0   0      0  4620564 255616     -        0   8.1  1.5 /Volumes/vm/workspace/go/src/github.com/pion/ion/bin/sfu -c /Volumes/

[kngako]

What seems to be happening on my end is that keeping a solo stream open for around 10 minutes causes the memory usage of sfu to increase by like 200mb (from an initial 15mb). Even after a stream is closed this memory is never released until the process gets killed (by the user or by the OS, I think it once got killed while I had a friend on the stream with me around the 17:30 mark in the graph below). Don't know what is getting allocated memory but not getting released though. Don't know if looking in the rtc Router.jitter would be the most fruitful approach.

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                            
29098 network   20   0 1541.2m 487.4m  17.0m S  3.8 49.5   2:33.44 ~/ion/bin/sfu -c ~/ion/configs/sfu.toml

Note: I'm running this on a 1GB ram droplet.

[tarrencev]

I think there are separate issues going on here. I ran into the original issue again, and it seems the sfu ect are still alive despite them being removed from islb service discovery. So I think the issue might be to do with the etcd key refresh

[adwpc]

This should be fixed.