dweb-camp-2019/meshnet

Mesh @ DWeb Camp 2019 📡

meshnet's Issues

Gigabit links through managed switch drops to 100M after a while

These are wireless wire links normally talking at 800 Mbps, passing thru Netgear gigabit managed switch for VLAN tagging, and into espressobin. Some of these links fall to 94 Mbps (100M) speeds all of a sudden and stay there. Tested with different cat5e cables. Suspect this is related to the Netgear switch. Rebooting it or entire node doesn't help. Same links have recorded 800+ Mbps.

Suspect: Netgear managed switch

• Switch setting
• Heat dissipation related
• Number of active ports
• Firmware

Raspbian Buster busted IP forwarding

Everything seems to work in newly released Raspbian Buster Lite except ifaces don't forward https://github.com/dweb-camp-2019/meshnet/blob/master/testnet/install#L18-L21

Workaround? Download Stretch Lite

Local NTP server

Many applications (such as monitoring) may require NTP to work properly. We have two options:

1. Have NTP through our Internet connection

2. @raylas suggested building our own NTP server on a Raspberry Pi

@benhylau thinks having a local time provider is an excellent idea, because it's a simple piece of hidden infrastructure that people can see and learn about.

Broadcast address mix-up on Linux Clients

Because I am using those 10.X.0.1/32 addresses for the mesh interfaces, I got into an interesting problem on client DHCPs.

Debug using: tcpdump -i lan port 67 or port 68 -e -n -v | grep "7f:98" -C 20

DHCP Request & Response:

20:18:57.747161 20:c9:d0:2c:7f:98 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17),
length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 20:c9:d0:2c:7f:98, length 300, xid 0x9cf5e867, Flags [none]
          Client-Ethernet-Address 20:c9:d0:2c:7f:98
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Request
            Requested-IP Option 50, length 4: 10.60.30.107
            Hostname Option 12, length 5: "white"
            Parameter-Request Option 55, length 13:
              Subnet-Mask, BR, Time-Zone, Default-Gateway
              Domain-Name, Domain-Name-Server, Option 119, Hostname
              Netbios-Name-Server, Netbios-Scope, MTU, Classless-Static-Route
              NTP
20:18:57.771874 ce:d6:2b:ac:f4:b0 > 20:c9:d0:2c:7f:98, ethertype IPv4 (0x0800), length 342: (tos 0xc0, ttl 64, id 1010, offset 0, flags [none], proto UDP (17)
, length 328)
    10.60.0.1.67 > 10.60.30.107.68: BOOTP/DHCP, Reply, length 300, xid 0x9cf5e867, Flags [none]
          Your-IP 10.60.30.107
          Server-IP 10.60.0.1
          Client-Ethernet-Address 20:c9:d0:2c:7f:98
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: ACK
            Server-ID Option 54, length 4: 10.60.0.1
            Lease-Time Option 51, length 4: 14400
            RN Option 58, length 4: 7200
            RB Option 59, length 4: 12600
            Subnet-Mask Option 1, length 4: 255.255.0.0
            BR Option 28, length 4: 10.60.0.1
            Default-Gateway Option 3, length 4: 10.60.0.1
            Domain-Name-Server Option 6, length 4: 10.60.0.1
            Hostname Option 12, length 5: "white"

Because of the /32 set on some interfaces, dnsmasq decides to send:

BR Option 28, length 4: 10.60.0.1

which makes sense. What doesn't make sense is the Mac doesn't care and sets broadcast address to 10.60.255.255 while the Linux respects 10.60.0.1 and doesn't route at all :D

The fix f651e1b is easy, but the platform variation is interesting.

DHCP Response (fixed):

20:34:20.412199 ce:d6:2b:ac:f4:b0 > 20:c9:d0:2c:7f:98, ethertype IPv4 (0x0800), length 342: (tos 0xc0, ttl 64, id 50744, offset 0, flags [none], proto UDP (17), length 328)
    10.60.0.1.67 > 10.60.30.107.68: BOOTP/DHCP, Reply, length 300, xid 0x26940e25, Flags [none]
          Your-IP 10.60.30.107
          Server-IP 10.60.0.1
          Client-Ethernet-Address 20:c9:d0:2c:7f:98
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: ACK
            Server-ID Option 54, length 4: 10.60.0.1
            Lease-Time Option 51, length 4: 14400
            RN Option 58, length 4: 7200
            RB Option 59, length 4: 12600
            Subnet-Mask Option 1, length 4: 255.255.0.0
            BR Option 28, length 4: 10.60.255.255
            Default-Gateway Option 3, length 4: 10.60.0.1
            Domain-Name-Server Option 6, length 4: 10.60.0.1
            Hostname Option 12, length 5: "white"

Gigabit PoE equipment

Thinking about PoE for the single-board computers to minimize the number of power cables. Gigabit equipment with 802.3af support is probably more useful overall, even though Raspberry Pis are 10/100 at the moment.

Switches

• e.g. https://www.canadacomputers.com/product_info.php?cPath=569_27_1045_346&item_id=115540
• Ubiquiti TOUGHswitch and other Ubiquiti equipment are passive and not 802.3af
• @jonah-archive does IA have 802.3af 10/100 switches we can use?

Splitter

• e.g. https://www.amazon.com/dp/B07CNKX14C/

Update espressobin documentations

Add docs for remote access to radio web interfaces

Some commands for finding radios and proxying web interface:

sudo arp-scan -l
sudo arp-scan -l --interface=en5
ip neigh
ip -6 neigh
arp -a
socat tcp-listen:8000,reuseaddr,fork tcp:192.168.88.1:80

List of networking accessories

This issue collects a list of networking accessories that are needed.

The top message will be updated as we come up with more items. Please add ideas to the thread.

---

• NEMA-rated box like this one
• Spool of ethernet cables (how much of this? For core infra + workshop to crimp cables)
• Ethernet cable crimper kits
• Equipment wall/pole mounts (perhaps @raylas can suggest?)

Monitoring the mesh

As discussed earlier with @raylas we'd like a way to monitor the network. Now we have a pretty good idea of topology #2 (comment) does anyone want to bring a collection server and work together on telling our nodes to push metrics to it? The interface should be pretty clean because we only have 6-8 mesh nodes all with routable IPs and no NAT.

---

Is the group planning on collecting data and visualizing these metrics?

• SNMP based monitoring platforms - LibreNMS, PRTG
• Input data into InfluxDB for data retrieval
• Dashboards like Grafana, Kabana, etc.

We are imagining pushing metrics to InfluxDB, where SNMP monitoring tools can populate at a preconfigured endpoint. SNMP is going to be readily available by most vendors of systems/networking equipment. For those devices/metrics that aren't SNMP-ready, we can use a series of distributed scripts populating it.

I think this is an important piece we need to plan, not only as network operators but to also map it to physical topologies so participants of the Camp can discuss and map the digital world to a self-contained network we can visually see its infrastructures.

Call for Network Stewards

Network Stewards at DWeb Camp are people who are committed to helping other campers learn about and tinker with the local network and Internet connectivity arrangement, and to help people address general device connectivity issues should that arise around them during their stay at Camp. Here are the expectations:

• Have general familiarity with the local network infrastructure. This does not imply technical proficiency, but it means:
  • Have spent an hour looking through documentation on this repository and its Issues board
  • Have gone on a short 15 minute tour to physically locate the wireless mesh network nodes and where local services are provided, and can point these infrastructures out to other campers or lead such a 15 minute walk to onboard new Network Stewards
• Have joined this Matrix chat room so they can ping other Network Stewards during Camp as needed
• Wear signage to indicate they are here to help if people are struggling with network issues or would like to learn more
• Open to leading/helping activities relating to the network infrastructure at Camp

Some but not all of the Network Stewards would have:

• Participated in the build events leading up to Camp
• Technical familiarity with the network topology
• Full admin access to the network infrastructure

We hope to have at least 12 Network Stewards during Camp days. Please post on this thread or say hi on the Matrix chat if you are interested to take part!

Misc items for Camp

Items to purchase/bring:

• Button supplies for 1" button maker
  • Pinbacks
  • Plastic rounds
  • Circle metal backings
• Pushpins and threads for mapping nodes
• Power adapters for cooling fans
• Mash potatoes for Mesh Lunches
• Paper cups for activity
• Short ethernet cables

Items for printing/signage:

• RJ45 pinouts on 1" buttons with this template
• Signage Network Orientation
• Cheatsheet for Network Stewards

How to DNS?

I would be excited if people try out radical DNS solutions like dnssb, but perhaps we need some centralized DNS on site? I don't know... what do people think?

How to get a SSL cert for your locally hosted app

Meshnet questions

Moved from dweb-camp-2019/organizing#17

This issue collects a list of questions that the prototyping process needs to answer.

The top message will be updated as more questions emerge that I don't currently have an answer for. Please add questions to the thread.

---

• Will/should I be able to host a service on the network from my laptop that is connected to the network? Or will I need to bring a microcomputer (e.g. rpi) and plug into one of these official planned nodes? #2 (comment)

• Will their be an on-site DNS server that's consistently managed for internal FQDNs? Or, will there be extensive use of mDNS implementations like Avahi? #2 (comment)

• Will multicast UDP be supported? If so, it could be an easy way to do service discovery across the network using Multicast DNS #2 (comment)

• How do I provide a web service to attendees via the mesh network? #2 (comment)

• What kind of work can I do ahead of time so minimize amount of hacking I need to do at the event? (i.e. do I need to bring my own device? Does that device need to have an Ethernet port or WiFi radio? Will there be 'golden servers' that are preconnected and I should bring a USB driver with software in a certain format like a docker image or tarball?) #2 (comment)

Update OmniTIK documentations + scripts

Review meshnet hardware used by existing communities

Moved from https://github.com/dweb-camp-2019/organizing/issues/8

What hardware is used by mesh communities and how are they using them:

• NYC Mesh
• People's Open
• Freifunk
• Guifi -- this one is more high level and related to backhaul than radio equipment
• Altermundi / LibreRouter
• LibreMesh

Configure Raspberry Pi for 10.8.8.8

• Implement setting nodeid in install script (so we can set node-10)
• Run ip-dir
• Shared HDD
• Possibly as DNS server

Add deployment notes

Add interesting failure modes during deployment days (e.g. partially functional node with SD card popped out)

Get budget estimate to Wendy for budgeting

Update wAP documentations + script

List of potentially useful hardware

Tracking a list of hardware that may be useful for us. Please include the following details:

• Name: device model
• Description: what is it and why it may be useful
• Availability: pricing information, where to get it, will you bring them?

Post-DWeb Camp 2019 mesh equipment inventory

Item	Internet Archive
MikroTik Wireless Wire (pair)	5
MikroTik SXTsq 5 ac	8
MikroTik mANTBox 15s	4
MikroTik OmniTIK 5 PoE ac	4
MikroTik wAP ac 3x3 (black)	3
MikroTik cAP ac	1
MikroTik QMP mount	2
Ubiquiti UBAM mount	2
Cat5e connectors (? of 100 pcs)	1
ESPRESSObin V5 64 Bit	6 + 2 damaged
ESPRESSObin V7 64 Bit	1
ESPRESSOBIN+ power supply	8
NETGEAR 5-Port Gigabit Managed Switch	4 + 1 damaged
Outdoor Junction Box	6
UCTRONICS PoE Splitter (Gigabit 802.3af Micro-USB 5V)	15
Raspberry Pi 3B+	4 (from Summit 2018)
Cooling fans (for outdoor nodes)	4

Network activities planning

• Draft Starter Kit to set expectations (@GoodieHicks)
• Draft Challenges based on our deployment experience (@GoodieHicks)
• Map of physical locations to node-ids (@GoodieHicks)
• Print large maps and make buttons (@GoodieHicks + @benhylau)
• Start email thread with Network Stewards (@benhylau)
• Ensure Network Stewards are in same Matrix chat (@benhylau)
• Submit activities to SCHED (@benhylau)

Forum

SD card corruption on poweroff without shutdown

Same issue as all other SBC systems where SD card may get corrupt on hard power off.

Plan:

1. SSH in to shutdown before powering off
2. Create a node with known ID as replacement SD

Espressobin hospital

Two used espressobins arrived dead, and one that we've been using seems to have a broken wan port. They are now labeled and this sheet tracks statuses at the mesh node hospital:

Node	Description	Status
P1	Purchased used, never worked	Doesn't boot
P2	Purchased used, never worked	Doesn't boot
P3	Purchased new, probably worked before	wan port seems dead

Review meshnet hardware used by existing communities

What hardware is used by mesh communities and how are they using them:

• NYC Mesh
• People's Open
• Freifunk
• Guifi -- this one is more high level and related to backhaul than radio equipment
• Altermundi / LibreRouter
• LibreMesh

Do we need IPv4?

We should discuss whether we need IPv4, and if so, how to provide it on the local network :)

But then I am also mindful apps don't like IPv6 still, so... They kinda need the v4 nat
-- @benhylau

I almost just wanna run a yggdrasil ipv6 only network and let apps that don't support v6 fail catastrophically
-- @benhylau

I also like the idea of pushing to have the network be IPv6-only! Fun fact: T-Mobile's network infrastructure (including cellular) is IPv6 only and uses v4-in-v6 tunneling when needed.
-- @raylas

Mesh links formed by wireless wires iperf3 asymmetric speeds

iperf3 -c PEER_IP from one espressobin node to another gets 350 Mbps, while it reaches 700+ Mbps with the -R flag. Unsure where these halvings come from.