helium/helium-packet-router

helium-packet-router's Issues

Regional Chirpstack support

Bypass State Channels

Use GRPC box client instead of GRPC client

Get feedback from gateway on downlink (too early, to late)

Packet Tracking via S3

Periodically send packet tracking info to S3

Change GRPC maps to records now that we only use GRPC box

• #109
• #115

Create Debian package for HPR

See PR#17

Tasks:

• construct Debian package .deb file based upon a successful rebar3 release run
  • Used router repo as placeholder: rebar3 as testnet rel;
  • See helium/router#847
• confirmed .deb generation against router repo; see Draft PR
• authored systemd config and launch scripts (required)
• authored /etc/init/ and /etc/rc?.d/ launch scripts
• authored Dockerfile for testing within a pristine container
• authored Vagrantfile for testing within a pristine VM
• confirmed .deb extraction within clean Docker image
• confirmed .deb extraction within clean Vagrant VM
• confirmed successful deploy of .deb within clean container
  • Blocked on Helium packet-router main branch having successful make rel
• confirmed successful deploy of .deb within clean VM
  • Blocked on Helium packet-router main branch having successful make rel

Future efforts:

• Local repo for storage until accepted into a Debian repo
• Successful submission to official Debian repo
  • For a rough idea of Debian package submission complexity:
    https://www.debian.org/doc/manuals/packaging-tutorial/packaging-tutorial.en.pdf#Outline0.6.1.42

OUTDATED sketch:

Debian packages can be generated via Makefile, and here's a generic version that just needs to be confirmed against current man pages and refined:

# Excerpt for Makefile to build .deb package

# FIXME: don't use var in this way.  this is just to highlight it in this draft:
MAIN_THING=helium-packet-router

RELEASE=0.1.0
PACKAGE_BUILD=`date +%Y%m%d.%H%M%S`
REVISION=${RELEASE}-${PACKAGE_BUILD}
DEBIAN_PACKAGE_NAME=helium-packet-router

DESTDIR=/opt/hpr

RUNTIME_USER=nobody

package: package-bin package-cfg


# FIXME: change names of .sh scripts
release: ${MAIN_THING}
	rm -rf release
	mkdir -p release${DESTDIR}/{bin,var/{log,run}}
	cp -p ${MAIN_THING}{,.ini,.sh} release${DESTDIR}/bin/

# FIXME: update to accommodate systemd
package-bin: release
	mkdir -p release/DEBIAN
	find release -type d | xargs chmod 755
	sed "s/Version: 1.0-1/Version: ${REVISION}/" < deb-bin-control.txt | sed "s/Architecture: all/Architecture: `dpkg --print-architecture`/" > release/DEBIAN/control
	svn info | egrep 'URL|Revision' | sed 's/^/ /' >> release/DEBIAN/control
	echo "#! /bin/sh" > release/DEBIAN/postinst
	echo 'chown -R `((id ${RUNTIME_USER} >/dev/null && echo ${RUNTIME_USER}:${RUNTIME_USER})|| echo nobody:nogroup)`' ${DESTDIR}/ >> release/DEBIAN/postinst
	echo "ln -s ${DESTDIR}/bin/${MAIN_THING}.sh /etc/init.d/" >> release/DEBIAN/postinst
	echo "ln -s ${DESTDIR}/bin/${MAIN_THING}.sh /etc/rc3.d/S40${MAIN_THING}" >> release/DEBIAN/postinst
	echo "ln -s ${DESTDIR}/bin/${MAIN_THING}.sh /etc/rc1.d/K40${MAIN_THING}" >> release/DEBIAN/postinst
	echo "echo To manually start, use:" >> release/DEBIAN/postinst
	echo "echo su - ${RUNTIME_USER} ${DESTDIR}/bin/${MAIN_THING}.sh start" >> release/DEBIAN/postinst
	echo "#! /bin/sh" > release/DEBIAN/prerm
	echo "${DESTDIR}/bin/${MAIN_THING}.sh stop" >> release/DEBIAN/prerm
	echo "${DESTDIR}/bin/${MAIN_THING}.sh unkind-stop" >> release/DEBIAN/prerm
	echo "${DESTDIR}/bin/${MAIN_THING}.sh clean-logs" >> release/DEBIAN/prerm
	echo "rm -f ${DESTDIR}/var/run/scoreboard.txt*" >> release/DEBIAN/prerm
	echo "rm -f ${DESTDIR}/var/log/console.log.[A-Z]*" >> release/DEBIAN/prerm
	echo "rm -f /etc/init.d/${MAIN_THING}.sh" >> release/DEBIAN/prerm
	echo "rm -f /etc/rc?.d/*${MAIN_THING}" >> release/DEBIAN/prerm
	chmod 755 release/DEBIAN/{postinst,prerm}
	dpkg-deb --build release
	mv release.deb ${DEBIAN_PACKAGE_NAME}-bin_${REVISION}_`arch`.deb
	ls -lh *.deb

package-cfg: ${MAIN_THING}_${RELEASE}.ini
	mkdir -p release-config/DEBIAN
	mkdir -p release-config${DESTDIR}/bin/
	find release-config -type d | xargs chmod 755
	cp -p ${MAIN_THING}_${RELEASE}.ini release-config${DESTDIR}/bin/${MAIN_THING}.ini
	sed "s/Version: 1.0-1/Version: ${REVISION}/" < deb-cfg-control.txt > release-config/DEBIAN/control
	svn info | egrep 'URL|Revision' | sed 's/^/ /' >> release/DEBIAN/control
	echo "#! /bin/sh" > release-config/DEBIAN/postinst
	echo "${DESTDIR}/bin/${MAIN_THING}.sh reload-config" >> release-config/DEBIAN/postinst
	chmod 755 release-config/DEBIAN/postinst
	dpkg-deb --build release-config
	mv release-config.deb ${DEBIAN_PACKAGE_NAME}-cfg-DEL_${REVISION}.deb
	ls -lh *.deb

clean:
	rm -rf release release-config *.deb

Chirpstack multi-region support

Chirpstack Application Server V4 supports multiple Regions through the use of multiple Gateway bridges.

All of these gateway bridges when deployed have different ports, Configurations only know about 1 endpoint, and should continue to do so.

How do we support different regions going to different endpoints without needing to specifically support it in the config service.

Match GWMP (or other protocols) gateway ID to Helium pub key and name for better debugging

Logging sweep

Packet Tracking transition

HPR need to have an HTTP port opened to ingest data above and then send to S3. We should only allow keys from known OUIs

GRPC packet ingestion

Helium gateways speak gRPC to Helium LoRaWAN Network Servers (LNS)

• #10
  • GitHub-CI is successful with grpc builds
• Define new protobuf: (Thanks, Pierre!) See packet_router.proto in helium/proto#157
• Listen for inbound gRPC connections (#25)

EDIT: unnecessary former tasks or accommodated through other means:

• Tee raw gRPC traffic to local file:
  • Required as an input to our Packet Verify service (lives outside of HPR or LNS)
  • Accommodates replaying traffic for debugging
  • Accommodates live traffic going to next Release Candidate but with all safeguards of Production environment
• Forward to router as gRPC (#30)
• Accommodate load-balancer sessions with id/token in gRPC values or headers

Notes:

• Inbound traffic from gateway-rs will be gRPC flowing through a load-balancer
  • AWS app layer ELB is gRPC aware
• Handling of large scale traffic volumes will be deferred to load-balancer
  • e.g., Let LB maintain large socket listen queue because setup/shutdown of sockets from LB to packet-router across AWS backbone should be relatively cheap
  • Therefore, nothing special to be done for tuning Linux kernel or Erlang/BEAM
• This ticket is just for the Listening portion of gRPC
  • The companion use of http2_client for accommodating sending to Class C devices (downlink without an uplink initiating the request) will be implemented as a separate effort

IoT Packet Verifier MVP

Build an initial simple implementation of IoT Packet Verifier based upon #34

• Compile using HPR's draft Protobufs and decode into Rust struct:
  • New branch for PV protos helium/proto#171
  • Branched from Pierre's helium/proto#157
  • HPR protos merged into Pierre's helium/proto#162
• Read from S3 using FileStore (e.g., file_sink.rs)
• Write to S3 using FileStore
• Read NetID/OUI mappings from Config Server
• Tests passing

Notes:

• It's called verifier (not validator) to avoid confusion with other systems currently within The Helium Network
• This is version zero (not v1 yet)

Metrics

Intermittent eunit and ct test failures

%%% hpr_routing_SUITE ==> 
Failure/Error: ?assertEqual({25,1}, receive_gateway_limit_exceeded_test ( { 0 , 0 } ))
  expected: {25,1}
       got: {26,0}
      line: 85```

```eunit --module hpr_max_copies```
I lost the example of the eunit test when I re-ran it and it succeeded.

Investigate stream half closed

HTTP roaming support

Support routing of packets to roaming partners via HTTP.
Passive routing only will be supported, based on the function and implementation of roaming in Packet Purchaser.

Check all CTs and run them individually in GitHub actions

Agregate logs (aka Loki)

Investigate Loki

Add hotspot metadata in uplinks

We will need to add hotspot name in our uplinks & join accepts so that Chirpstack can display it in LoRaWAN frames & Events.

https://github.com/chirpstack/chirpstack-gateway-bridge/blob/master/internal/backend/semtechudp/packets/push_data.go#L345

Get and save Routing Config from Config Service

Basic Auth header for HTTP roaming

Support MIC checking

• Subscribe to update from HCS
• If a session key is assigned to a DevAddr we should check the computed MIC

See helium/helium-config-service#21

Dockerizing

Failed packet delivery to one route should not prevent delivery to remaining routes

hpr_routing:deliver_packet/3

Support downlink RPC from HCS

See helium/helium-config-service#22

CI

Create boiler-plate config for gRPC and protobufs

Create boiler-plate config for gRPC and protobufs for helium-packet-router:

• Configuration for gRPC and protobufs replicated from various other Helium repos:
  • e.g., blockchain-core, router, miner, sibyl
  • Followed Mack's excellent trail of breadcrumbs through those repos
• Since helium-packet-router is more "gateway" than "router", various tweaks were made to this instance of the config
• Primary differences from router are:
  • Behaviour here reuses helium_router_bhvr, of which its implementation in router was recently updated
  • Here, we'll just use that one rpc's StateChannel param as an envelope around Packet
  • Considered but ruled-out using helium_gateway_bhvr due to rpc end points not needed here
  • See src/grpc/helium_packet_router_service.erl
• Notes regarding the Erlang "behaviour"
  • hpr doesn't need anything to do with state-channels
  • but some of that info already being sent by gateway-rs may be useful metadata or for logging
  • hpr probably should ignore (pass-through) anything region-specific
  • etc. (please comment on this ticket if you can think of any others)
• See PR#9

Tests for GWMP worker

Port relevant tests from packet-purchaser.

LNS should have a way to give feedback about packet

If packet is not deemed worthy by the LNS it should not be enter in the s3 bucket

Investigate GRPC box client

The goal here is to only have to support one library for client/server.

Use new proto to register gateway to server

Route packet based on Config

fix hpr_http_roaming_downlink_handler:send_response

Use hpr_packet_service:packet_down/2.

Bring multi packet to HPR

#74

Debug Tools for HPR

Add payload size to packet reports

MIC Benchmark

We need to know an approximate limit on how many "MIC brute forcing" we can do before seeing a significant slow down.

Here is the Router code that does the MIC calculation.

Add location based filtering

• Hotspot send it's h3 index

• Route can include "no go zones" where it does not want to get data from

• If hotspot in zone packet is not forwarded

• h3 index should be sent to Packet verifier adding an extra check

• what do we do for data only?

Add backoff on to config connection

Improve efficiency of ets match spec

helium-packet-router/src/hpr_routing_config_worker.erl

Line 68 in ae7306c

{{'$1', '$2', '$3'}, '$4'},

The match spec can probably be made more efficient by using the key value in the matching tuple instead of checking the value in the guard, something like:

            MS = [
                {
                    {{NetID, '$2', '$3'}, '$4'},
                    [
                            {'andalso', {'=<', '$2', DevAddr}, {'=<', DevAddr, '$3'}}
                    ],
                    ['$4']
                }
            ],

This allows ETS to take advantage of the key column.

Use current GRPC router service to send packet to Router

Here is an example of client side grpc (more in the repo)

https://github.com/helium/sibyl/blob/master/config/grpc_client_gen.config

For ref the server side is here https://github.com/helium/router/blob/master/src/grpc/helium_router_service.erl

Use AWS S3 Roles to connect to S3 instead of key/secret

• Add Role to Alpha Server to allow access to S3 bucket
• Check if AWSD library support keyless access

Update GRPC definition

roaming on dataonly hotspots

Hi, i was using Senet devices roaming on a data-only hotspot: it stopped working in June, confirmed by Helium staff here. any plans to enable roaming again? please advise if i should post elsewhere, thanks.

Implement metrics

Implement GWMP

Routing Config based on Proto

• Create proto route matching

{
        net_id=3,
        dev_ranges=[{12, 20}, ..]
        euis=[{appeui, deveui}, ...] 
        lns_address="lora.com|10.0.0.9"
        protol="gwmp|http"
        oui=1
}

• Save config locally

Include Downlink Service

Packet Verifier design docs

IoT Packet Verifier will be our service inspecting LoRaWAN packets and associated metadata and will validate a Gateway’s stream of Uplinks in part by also inspecting Downlinks.

• Draft simple document for design criteria: Michael & Daniel
• Research / Explore protobufs for LoRaWAN bits within ChirpStack (aka don't reinvent the wheel)
• Learn about FileStore and file_sink.rs from poc5g code (Thank you, madninja!)
• Draft of simple design doc: Daniel
  • per Madninja's revised criteria: get to parity with existing router billing functionality
  • This is an MVP; i.e., v0 (rather than `v1)

Next will be: MVP implemented in Rust; see #43

This is separate from the Payment System but related. One feeds the other and back again.