Converting of AIS length/beam could produce errors

NMEA2000-Gateway with ESP32

Based on the work of

Homberger
Timo Lappalainen
Arno Duvenhage and a couple of other open source projects. Many thanks for all the great work.

This project is part of OpenBoatProjects.

Important Hint

Although the term NMEA2000 is used here, the software or device is not a certified NMEA2000 device as it did not pass any approval process. There are chances that the software / device does not follow the NMEA2000 specification at several points. If you connect the device to your NMEA2000 network you do this on your own risk.

Goal

Have a simple ready-to-go ESP32 binary that can be flashed onto a M5 Atom CAN, potentially extended by an Atom Tail485 for NMEA0183 connection and power supply.

But will also run on other ESP32 boards see Hardware.

What is included

a NMEA2000 to Wifi (NMEA0183) gateway
a NMEA2000 to USB (NMEA0183) gateway
a NMEA0183 Multiplexer
- Wifi to Wifi
- Wifi to USB
- Wifi to RS485
- and reverse
a NMEA0183 RS485 to NMEA2000 gateway
a Wifi (NMEA0183) to NMEA2000 gateway
an USB (NMEA0183) to NMEA2000 gateway
a WEB UI to configure the gateway and to show the data that has been received
a USB Actisense to NMEA2000 gateway
a NMEA2000 to USB Actisense gateway
starting with 201311xx some I2C Sensors
starting with 20240324 SSI rotary encoders

For the details of the mapped PGNs and NMEA sentences refer to Conversions.

Hardware

The software is prepared to run on different kinds of ESP32 based modules and accessoirs. For some of them prebuild binaries are available that only need to be flashed, others would require to add some definitions of the used PINs and features and to build the binary. For the list of hardware set ups refer to Hardware.

There is a couple of prebuild binaries that you can directly flash to your device. For other combinations of hardware there is an online build service that will allow you to select your hardware and trigger a build.

Installation

In the release section you can find a couple of pre-build binaries.
They are devided into binaries for an initial flash (xxx-all.bin) and binaries for updating an existing device (xxx-update.bin).

For other Hardware refer to the online build service.

Initial Flash

Browser

If you run a system with a modern Chrome or Edge Browser you can directly flash your device from within the browser. Just go to the Flash Page and select the "Initial" flash for your Hardware. This will install the most current software to your device. If you are on Windows you will need to have the correct driver installed before (see below at windows users - only install the driver, not the flashtool).

You can also install an update from the flash page but normally it is easier to do this from the Web Gui of the device (see below).

The Flash Page will also allow you to open a console window to your ESP32.

Tool based

To initially flash a deviceyou can use ESPTool. The flash command must be (example for m5stack-atom):

esptool.py --port XXXX --chip esp32 write_flash 0x1000  m5stack-atom-20211217-all.bin

For the meaning of the board names have a look at Hardware. For details refer to the code in platformio.ini and look for the hardware definitions in GwHardware.h. Additionally there is a small GUI for the esptool included here at tools/flashtool/flashtool.py

linux users
You can typically install the esptool (once you have python 3 installed) with

sudo pip install esptool

To use the flashtool just download flashtool.pyz.

sudo pip install tkinter
sudo pip install pyserial

Afterwards run flashtool.pyz with

python3 flashtool.pyz

windows users
You need to install the driver for the serial port to connect your ESP32 board. For a modern windows the driver at FTDI should be working. After installing the driver check with your device manager for the com port that is assigned to your connected esp device.

For the flashtool you can find a prebuild executable in tools: esptool.exe. Just create an empty directory on your machine, download the esptool to this directory and also download the binary (xxx-all.bin) from releases.

Open a command prompt and change into the directory you downloaded the esptool.exe and the firmware binary. Flash with the command

esptool.exe --port COM3 write_flash 0x1000 xxxxx-xxxx-all.bin

Replace COM3 with the port shown in the device manager and the xxx with the name of the downloaded binary.

If you do not want to use the command line there is a tool with an UI that allows you to flash the initial or update firmware. Just download the exe for your windows system from the ESP32N2K-Flasher Git Repository. There is no installation needed - just start the downloaded exe. Some Anti Virus Software may (accidently) tag this as infected. In this case you can still install the UI in two steps:

you first need to install python3 from the download page - use the Windows 64 Bit installer. Install using the default settings.
Afterwards download flashtool.pyz and run it with a double click.

Update

To update a device you can use the Web-UI (Update tab). In principle you could also update a device using the initial flash command (and an xxx-all.bin) firmware but this would erase all your configuration. So for normal operation just download a xxx-update.bin from the release page and use the UI to install it.

.

When you choose a file for the update the UI will check if it is a valid firmware file and will reject invalid ones. To really execute the update click the "Upload" button. You will have a progress indicator and get a notification about the update result. Please reload the page in your browser after the "connected" state is green as the new version could have changes thatv otherwise will not work.

Starting

After flushing a wifi access point is created. Connect to it (name: ESP32NMEA2K, password: esp32nmea2k). Afterwards use a Bonjour Browser, the address ESP32NMEA2k.local or the ip address 192.168.15.1 to connect with your browser. You will get a small UI to watch the status and make settings. If you want to connect to another wifi network, just enter the credentials in the wifi client tab and enable the wifi client. For all the potential inputs and outputs (NMEA2000, USB, TCP, RS485) you can set the configuration including NMEA0183 filtering. To store your changes you will be asked for an admin password. The initial one is esp32admin. You can change this password at the config/system tab (and even completely disable it). Be careful to notice the password - you can only recover from a lost password with a factory reset of the device (long press the led button until it goes blue->red->green). On the data page you will have a small dashboard for the currently received data. On the status page you can check the number of messages flowing in and out. To help you recover lost passwords the Wifi access point passowrd and the admin password will be output at the USB port when the device starts up. So by connecting a terminal program you can retrieve those passwords.

Security Hints

You should only connect the Wifi of the device to trusted networks. There is only some very limited protection against network sniffing of denial of service attacks. Never connect the device directly to the internet without a firewall in between (like e.g. your Wifi or LTE router). Especially be careful when connecting to open port networks. When making changes you will be asked for the admin password - and this one is always send somehow encrypted. But when you change the Wifi access point password or the Wifi client password it will be sent in clear text. When you enable the "remember me" for the admin password it will be stored in clear text in your browser (use ForgetPassword to remove it from there).

Conversion from and to NMEA0183 XDR

The gateway can convert a lot of NMEA 2000 PGNs that have no direct NMEA0183 equivalent to NMEA0183 XDR records. For details refer to XdrMappings

Development Environment

PlatformIO.

Extending the Software

To give room for adding own software and still being able to keep in sync with this master part there is a concept of user tasks that will allow you to add your own hardware definitions and to add code that should be executed without the need to change parts of the existing software. For details refer to the example description.

Changelog

20240428

fix build error with M5 gps kit 20240324

add SSI rotary encoders
add some options to the converter (RMC rate, RSA parameters)
support for the M5 Atomic PortABC - more grove ports #58
some restructuring in the hardware definitions
add SSID to status page #37
allow to attach i2c sensors to the grove ports in the build service
add calset and calval config types
This requires to write current values of a sensor at the api with setCalibrationValue. The user can then bring up a calibration dialog and can set the current value as the config value.
change log flushing to USB port that prevented ESP32 S3 based boards to start if no USB connection was available
prevent the Web UI from appearing frozen if there was a large amount of invalid NMEA data received #60

20231228

lock AsyncTCP-esphome to 2.0.1 to avoid compile errors
own main loop to avoid deadlocks with serial send in user tasks

20231105

support for ESP32-S3
own TWAI based driver for the NMEA2000 bus
add NMEA2000 node address and status to the status tab
ability to change the AP ip address
online build service to select the components you need
restructuring of the lib_deps handling (much shorter compile time
Hint: if this introduces problems for your build, revert back the lib_ldf_mode to chain+
integration of a couple of I2C sensors (e.g. M5 ENVIII, BME280)
More functionality for user tasks (counter, interface between tasks, dynamic registration, adding fixed XDR mappings) - refer to the example description

20230317

correctly convert bar to Pascal in XDR records

20230309

use underscores in settings file names #40
pin platform and lib versions
add rs232 and rs485 atom boards
use less memory when saving new config
correct factor for ROT #44
better handling of VHW - send STW (128259) even if no heading, additionally send 127250 (magnetic/true) if included in VHW #49
parse MTW and convert to 130310 #49

20220403

add support for PGN 127257 pitch/roll/yaw
only convert RMC and GGA if valid status #38
only send 129539 if GSA fix mode is 2 or 3 #39

20220320

accept lowercase characters in NMEA0183 checksum #33
only show apPassword in UI if change is allowed #34
optimize memory usage for set config values
add a source field to GwApi::BoatValue
make NMEA0183 messages robust against to much fields
reset the can bus driver if the queue remains full for 2s (avoid problems if initially no other device is at the bus)

20220308

correctly handle select fields when importing the config
better names for config save #26
remove -e in ci build #30

20220302

add export and import of config data.
HINT: passwords are not included
add a HELP tab that points to this description
The url can be changed using the HELP_URL capability

20220301

change boat data names to shorter ones
correct bug with boatData config items
allow for shorter conditions in config items (use arrays of allowed values for "or" condition)
add some static variables at BoatData to reference the known items (see exampletask)
make the known config names static members of GwConfigHandler
remove unused AWD boat data item

20220124

make the serial input and output working again 20220114

incorporate some changes from Homberger to improve AIS compatibility with Raymarine displays
introduce a global switch to prevent sending out converted NMEA2000 data
extension API improvements (hide config values, set config values)

20220112

correctly send out seasmart if NMEA out is not configured
enable TCP keepalive on connections to reconnect on failures 20220109

allow to set the log level in config

20220107

add a TCP client - you can connect to any source of NMEA data using IP address (or MDNS host name) and port
This way you can e.g. "chain" multiple gateways
add receiving of Seasmart messages.
Using this feature you can forward the data from the NMEA2000 bus via TCP to another device.
add some Status Info to the extension API
correct the display of wind direction on the data page

20211218