scrin/ruuvi-go-gateway

Software replica of the Ruuvi Gateway

ruuvi-go-gateway's Issues

Possible Race Condition? "concurrent map iteration and map write"

Greetings, thanks for making this!
I'm trying to use it to collect data from my Ruuvi sensors. I've attempted to run it overnight twice now and woken up to this panic.

./ruuvi-go-gateway 
fatal error: concurrent map iteration and map write

goroutine 6 [running]:
runtime.throw({0x7531e3, 0x45ebd9})
	/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0000bb870 sp=0xc0000bb840 pc=0x434951
runtime.mapiternext(0x19)
	/usr/local/go/src/runtime/map.go:858 +0x4eb fp=0xc0000bb8e0 sp=0xc0000bb870 pc=0x410ccb
reflect.mapiternext(0x95)
	/usr/local/go/src/runtime/map.go:1346 +0x19 fp=0xc0000bb8f8 sp=0xc0000bb8e0 pc=0x45eb19
reflect.(*MapIter).Next(0xc0002afe40)
	/usr/local/go/src/reflect/value.go:1618 +0x99 fp=0xc0000bb918 sp=0xc0000bb8f8 pc=0x485c79
encoding/json.mapEncoder.encode({0x6ddee0}, 0xc000036080, {0x6ef100, 0xc0005383b0, 0xc0002aef80}, {0x7, 0x0})
	/usr/local/go/src/encoding/json/encode.go:799 +0x305 fp=0xc0000bbad8 sp=0xc0000bb918 pc=0x514e05
encoding/json.mapEncoder.encode-fm(0x724700, {0x6ef100, 0xc0005383b0, 0x30}, {0x3c, 0x77})
	/usr/local/go/src/encoding/json/encode.go:779 +0x45 fp=0xc0000bbb18 sp=0xc0000bbad8 pc=0x51df45
encoding/json.structEncoder.encode({{{0xc0000cad80, 0xc0000bbc50, 0x4b1af9}, 0xc0001cf8c0}}, 0xc000036080, {0x724700, 0xc000538370, 0x7}, {0x0, 0x1})
	/usr/local/go/src/encoding/json/encode.go:761 +0x1f4 fp=0xc0000bbbc8 sp=0xc0000bbb18 pc=0x514854
encoding/json.structEncoder.encode-fm(0x6fc180, {0x724700, 0xc000538370, 0x475069}, {0xc0, 0xe2})
	/usr/local/go/src/encoding/json/encode.go:732 +0x69 fp=0xc0000bbc20 sp=0xc0000bbbc8 pc=0x51de89
encoding/json.structEncoder.encode({{{0xc0001962d0, 0xc0000bbd80, 0x4b7496}, 0xc0001cf920}}, 0xc000036080, {0x6fc180, 0xc000538370, 0x0}, {0x0, 0x1})
	/usr/local/go/src/encoding/json/encode.go:761 +0x1f4 fp=0xc0000bbcd0 sp=0xc0000bbc20 pc=0x514854
encoding/json.structEncoder.encode-fm(0x6fc180, {0x6fc180, 0xc000538370, 0x700d80}, {0x0, 0x30})
	/usr/local/go/src/encoding/json/encode.go:732 +0x69 fp=0xc0000bbd28 sp=0xc0000bbcd0 pc=0x51de89
encoding/json.(*encodeState).reflectValue(0xc0000bbdc0, {0x6fc180, 0xc000538370, 0x40e2d4}, {0xc0, 0x8d})
	/usr/local/go/src/encoding/json/encode.go:360 +0x78 fp=0xc0000bbd88 sp=0xc0000bbd28 pc=0x5123f8
encoding/json.(*encodeState).marshal(0xc000538370, {0x6fc180, 0xc000538370}, {0x50, 0xbe})
	/usr/local/go/src/encoding/json/encode.go:332 +0xfa fp=0xc0000bbe00 sp=0xc0000bbd88 pc=0x51201a
encoding/json.Marshal({0x6fc180, 0xc000538370})
	/usr/local/go/src/encoding/json/encode.go:161 +0x45 fp=0xc0000bbe60 sp=0xc0000bbe00 pc=0x511805
github.com/Scrin/ruuvi-go-gateway/sender.SetupHTTP.func1()
	/github/workspace/sender/http.go:57 +0x1cf fp=0xc0000bbfe0 sp=0xc0000bbe60 pc=0x69b08f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000bbfe8 sp=0xc0000bbfe0 pc=0x4646a1
created by github.com/Scrin/ruuvi-go-gateway/sender.SetupHTTP
	/github/workspace/sender/http.go:46 +0x2e8

goroutine 1 [chan receive (nil chan), 870 minutes]:
github.com/go-ble/ble/linux.(*Device).Scan(0xc000021ae0, {0x7bcf30, 0xc00002c638}, 0x40, 0x0)
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/device.go:178 +0x8b
github.com/go-ble/ble.Scan({0x7bcf30, 0xc00002c638}, 0x70, 0xc000196240, 0x0)
	/go/pkg/mod/github.com/go-ble/[email protected]/gatt.go:92 +0x187
github.com/Scrin/ruuvi-go-gateway/gateway.Run({{0xc00002a330, 0x11}, 0x0, 0x0, 0xc0000c3730, 0xc00008c680, {{0xc00002ce60, 0x6}, {0xc00002ce78, 0x4}, ...}, ...})
	/github/workspace/gateway/gateway.go:67 +0x505
main.main()
	/github/workspace/cmd/ruuvi-go-gateway/main.go:33 +0x418

goroutine 7 [syscall]:
syscall.Syscall(0x0, 0x3, 0xc0001ac000, 0x1000)
	/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5
golang.org/x/sys/unix.read(0x44b0c5, {0xc0001ac000, 0x0, 0xc0000f4300})
	/go/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_linux.go:1215 +0x4d
golang.org/x/sys/unix.Read(...)
	/go/pkg/mod/golang.org/x/[email protected]/unix/syscall_unix.go:157
github.com/go-ble/ble/linux/hci/socket.(*Socket).Read(0xc000088680, {0xc0001ac000, 0x1000, 0xc0001ac000})
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/hci/socket/socket.go:123 +0x75
github.com/go-ble/ble/linux/hci.(*HCI).sktLoop(0xc0000b1040)
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/hci/hci.go:300 +0xc4
created by github.com/go-ble/ble/linux/hci.(*HCI).Init
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/hci/hci.go:153 +0x5f2

goroutine 8 [select, 870 minutes]:
github.com/go-ble/ble/linux/hci.(*HCI).Accept(0xc0000b1040)
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/hci/gap.go:185 +0xad
github.com/go-ble/ble/linux.loop(0x0, 0xc00008c8c0, 0x0)
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/device.go:55 +0x4a
created by github.com/go-ble/ble/linux.NewDeviceWithNameAndHandler
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/device.go:48 +0x1fd

goroutine 1754812 [select]:
net/http.(*persistConn).writeLoop(0xc00008b7a0)
	/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1748 +0x1e65

goroutine 1754811 [IO wait]:
internal/poll.runtime_pollWait(0x7f4a91a85798, 0x72)
	/usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc0008cc200, 0xc0005bc000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0008cc200, {0xc0005bc000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc0008cc200, {0xc0005bc000, 0x0, 0x0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020c040, {0xc0005bc000, 0xe, 0x0})
	/usr/local/go/src/net/net.go:183 +0x45
net/http.(*persistConn).Read(0xc00008b7a0, {0xc0005bc000, 0x447380, 0xc000217ec8})
	/usr/local/go/src/net/http/transport.go:1926 +0x4e
bufio.(*Reader).fill(0xc00020e5a0)
	/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).Peek(0xc00020e5a0, 0x1)
	/usr/local/go/src/bufio/bufio.go:139 +0x5d
net/http.(*persistConn).readLoop(0xc00008b7a0)
	/usr/local/go/src/net/http/transport.go:2087 +0x1ac
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1747 +0x1e05

Support TLS for MQTT connection

It would be cool if TLS connections for MQTT would be possible for running it over insecure (aka the Internet) networks.

Configuration option for Bluetooth device

The BLE scanner uses the system default device (https://github.com/Scrin/ruuvi-go-gateway/blob/master/gateway/gateway.go#L35).

In my case I have multiple bluetooth adapters and would like to configure which adapter to use for the gateway:

# hcitool dev
Devices:
        hci0    XX:XX:XX:XX:XX:XX
        hci1    YY:YY:YY:YY:YY:YY

Doesn't work with bookworm RPI3

I tested ruuvi-go-gateway with Debian 11 (bullseye), everything worked and then updated to Debian 12 (bookworm). hcitool/hcidump worked nicely but ruuvi-go-gateway didn't see any tags

dmesg contains

[ 1900.676659] Bluetooth: hci0: Opcode 0x c03 failed: -4
[ 1900.681386] Bluetooth: hci0: unexpected event for opcode 0x0c03

I managed to fix this by changing from go-ble/ble to rigado/ble and adding LEScanInterval + LEScanWindow:

  device, err := dev.NewDevice("default",
    ble.OptDeviceID(config.HciIndex),
    ble.OptScanParams(cmd.LESetScanParameters{
      LEScanType: 0, // passive scan
      LEScanInterval: 0x10,
      LEScanWindow: 0x10,
    }))

I guess go-ble doesn't validate those params and zero isn't working anymore.

Rate Limit

Would it be possible to add a configuration to set a rate limit for tag readings?

Maybe something like "only send readings if they are x seconds older than the last reading" (for any sensor/metric).

Feature Request - MQTT Will Message

Would it be possible to enhance the MQTT connection to send a Will Message on connection?

Ideally the message payload and topic could be set in the config file.

Thanks for considering.

No Ruuvitags detected

I have this running in docker on a Raspberry PI 4 using the sample docker compose file. I have 6 Ruuvitags nearby (all of which are detectable using the native Ruuvi iPhone app).

When I turn on trace level debug I can see lots of BLE advertisements but none are detected as being for the Ruuvitags. For example:

[2022-7-12 08:46:55] [trace] [/go/src/github.com/Scrin/ruuvi-go-gateway/gateway/gateway.go] Received data from BLE adapter, is_ruuvi: false, data: 4C0003161100000277C0A856200000000000000000000000000B, mac: 0D:72:48:9F:69:10, rssi: -59

[2022-7-12 08:46:55] [trace] [/go/src/github.com/Scrin/ruuvi-go-gateway/gateway/gateway.go] Received data from BLE adapter, is_ruuvi: false, data: 4C0002152686F39CBADA4658854AA62E7E5E8B8D000100000B, mac: 05:3E:AB:3B:0D:ED, rssi: -61

[2022-7-12 08:46:55] [trace] [/go/src/github.com/Scrin/ruuvi-go-gateway/gateway/gateway.go] Received data from BLE adapter, is_ruuvi: false, data: A705051001000000000000022200CA, mac: F3:33:A1:94:48:35, rssi: -66

So, I assume the BLE stack is working ok. If I change all_advertisements to true, I can see MQTT posts for the detected devices. So MQTT is working too. My tags are using data format 5 and running recent firmware.

Have you any idea why the gateway would see everything EXCEPT my Ruuvitags?