Version: 0.14.0-snapshot
The QNEthernet library provides Arduino-like Ethernet
functionality for the
Teensy 4.1. While it is mostly the same, there are a few key differences that
don't quite make it a drop-in replacement.
The low-level code is based on code by Paul Stoffregen, here:
https://github.com/PaulStoffregen/teensy41_ethernet
All the Teensy stuff is under the MIT license, so I'm making this library under the MIT license as well. Please see the lwip-info/ directory for the info files provided with the lwIP release.
- Differences, assumptions, and notes
- Additional functions and features not in the Arduino API
- How to run
- How to write data to connections
- A note on the examples
- A survey of how connections (aka
EthernetClient
) work - How to use multicast
- mDNS services
- DNS
- stdio
- Raw Ethernet frames
- How to implement VLAN tagging
- Other notes
- To do
- Code style
- References
This library mostly follows the Arduino Ethernet API, with these differences and notes:
- Include the
QNEthernet.h
header instead ofEthernet.h
. - Ethernet
loop()
is called fromyield()
. The functions that wait for timeouts rely on this. This also means that you must usedelay
,yield()
, orEthernet.loop()
when waiting on conditions; waiting without calling these functions will cause the TCP/IP stack to never refresh. Note that many of the I/O functions callloop()
so that there's less burden on the calling code. EthernetServer
write functions always return the write size requested. This is because different clients may behave differently.- The examples in https://www.arduino.cc/en/Reference/EthernetServerAccept and
https://www.arduino.cc/en/Reference/IfEthernetClient directly contradict each
other with regard to what
operator bool()
means inEthernetClient
. The first example uses it as "already connected", while the second uses it as "available to connect". "Connected" is the chosen concept, but different fromconnected()
in that it doesn't check for unread data. - All the Arduino-defined
Ethernet.begin
functions that use the MAC address are deprecated. - The following
Ethernet
functions are deprecated and do nothing or return some default value:hardwareStatus()
: ReturnsEthernetHardwareStatus::EthernetOtherHardware
because zero might be interpreted as "no hardware".init(uint8_t sspin)
: Does nothing.maintain()
: Returns zero.setRetransmissionCount(uint8_t number)
: Does nothing.setRetransmissionTimeout(uint16_t milliseconds)
: Does nothing.
- The following
Ethernet
functions are deprecated but call something equivalent:MACAddress(uint8_t mac[6])
setDnsServerIP(const IPAddress &dnsServerIP)
- The following
EthernetUDP
functions do nothing:flush()
because it is ill-defined.
- The system starts with the Teensy's actual MAC address. If you want to use
that address with the deprecated API, you can collect it with
Ethernet.macAddress
and then pass it to one of the deprecatedbegin
functions. - All classes and objects are in the
qindesign::network
namespace. This means you'll need to fully qualify any types. To avoid this, you could utilize ausing
directive:However, this pollutes the current namespace. An alternative is to choose something shorter. For example:using namespace qindesign::network; EthernetUDP udp; void setup() { Ethernet.begin(); }
namespace qn = qindesign::network; qn::EthernetUDP udp; void setup() { qn::Ethernet.begin(); }
- Files that configure lwIP for our system:
- src/sys_arch.c
- src/lwipopts.h โ Use this one for tuning (see src/lwip/opt.h for more details).
- src/arch/cc.h
- The main include file,
QNEthernet.h
, in addition to including theEthernet
instance, also includes the headers forEthernetClient
,EthernetServer
, andEthernetUDP
. - All four address-setting functions (IP, subnet mask, gateway, DNS) do nothing unless the system has been initialized.
QNEthernet defines functions that don't exist in the Arduino API as it's currently defined. (See: Arduino Ethernet library) This section documents those functions.
Features:
- The
read(buf, len)
functions allow a NULL buffer so that the caller can skip data without having to read into a buffer.
The Ethernet
object is the main Ethernet interface.
begin()
: Initializes the library and uses the Teensy's internal MAC address.begin(ipaddr, netmask, gw)
: Initializes the library, uses the Teensy's internal MAC address, and uses the given parameters for the network configuration.broadcastIP()
: Returns the broadcast IP address associated with the current local IP and subnet mask.end()
: Shuts down the library, including the Ethernet clocks.hostname()
: Gets the DHCP client hostname. An empty string means that no hostname is set. The default is "teensy-lwip".linkState()
: Returns abool
indicating the link state.linkSpeed()
: Returns the link speed in Mbps.linkIsFullDuplex()
: Returns whether the link is full duplex (true
) or half duplex (false
).joinGroup(ip)
: Joins a multicast group.leaveGroup(ip)
: Leaves a multicast group.macAddress(mac)
: Fills the 6-bytemac
array with the current MAC address. Note that the equivalent Arduino function isMACAddress(mac)
.setDNSServerIP(dnsServerIP)
: Sets the DNS server IP address. Note that the equivalent Arduino function issetDnsServerIP(dnsServerIP)
.setHostname(hostname)
: Sets the DHCP client hostname. The empty string will set the hostname to nothing. To use something other than the default at system start, call this before callingbegin
.waitForLocalIP(timeout)
: Waits for the specified timeout (milliseconds) for the system to have a local IP address. This is useful when waiting for a DHCP-assigned address. Returns whether the system obtained an address within the given timeout. Note that this also works for a static-assigned address.operator bool()
: Tests if Ethernet is initialized.- Callback functions: Note that callbacks should be registered before any other
Ethernet functions are called. This ensures that all events are captured. This
includes
Ethernet.begin(...)
.onLinkState(cb)
: The callback is called when the link changes state, for example when the Ethernet cable is unplugged.onAddressChanged(cb)
: The callback is called when any IP settings have changed.
static constexpr int maxMulticastGroups()
: Returns the maximum number of multicast groups.static constexpr size_t mtu()
: Returns the MTU.
close()
: Closes a connection, but without waiting. It's similar tostop()
.closeOutput()
: Shuts down the transmit side of the socket. This is a half close operation.writeFully(b)
: Writes a single byte.writeFully(s)
: Writes a string (const char *
).writeFully(s, size)
: Writes characters (const char *
).writeFully(buf, size)
: Writes a data buffer (const uint8_t *
).static constexpr int maxSockets()
: Returns the maximum number of TCP connections.
setNoDelay(flag)
: Sets or clears the TCP_NODELAY flag in order to disable or enable Nagle's algorithm, respectively. This must be changed for each new connection.isNoDelay()
: Returns whether the TCP_NODELAY flag is set for the current connection. Returns false if not connected.
begin(reuse)
: Similar tobegin()
, but the Booleanreuse
parameter controls the SO_REUSEADDR socket option.end()
: Shuts down the server.port()
: Returns the server's port.static constexpr int maxListeners()
: Returns the maximum number of TCP listeners.
begin(localPort, reuse)
: Similar tobegin(localPort)
, but the Booleanreuse
parameter controls the SO_REUSEADDR socket option.beginMulticast(ip, localPort, reuse)
: Similar tobeginMulticast(ip, localPort)
, but with areuse
parameter, similar to the above.data()
: Returns a pointer to the received packet data.localPort()
: Returns the port to which the socket is bound, or zero if it is not bound.send(host, port, data, len)
: Sends a packet without having to usebeginPacket()
,write()
, andendPacket()
. It causes less overhead. The host can be either an IP address or a hostname.static constexpr int maxSockets()
: Returns the maximum number of UDP sockets.
The EthernetFrame
object adds the ability to send and receive raw Ethernet
frames. It provides an API that is similar in feel to EthernetUDP
. Because,
like EthernetUDP
, it derives from Stream
, the Stream
API can be used to
read from a frame and the Print
API can be used to write to the frame.
beginFrame()
: Starts a new frame. New data can be added using thePrint
API. This is similar toEthernetUDP::beginPacket()
.beginFrame(dstAddr, srcAddr, typeOrLen)
: Starts a new frame and writes the given addresses and EtherType/length.beginVLANFrame(dstAddr, srcAddr, vlanInfo, typeOrLen)
: Starts a new VLAN-tagged frame and writes the given addresses, VLAN info, and EtherType/length.data()
: Returns a pointer to the received frame data.endFrame()
: Sends the frame. This returns whether the send was successful. A frame must have been started, its data length must be in the range 60-1518, and Ethernet must have been initialized. This is similar toEthernetUDP::endPacket()
.parseFrame()
: Checks if a new frame is available. This is similar toEthernetUDP::parseFrame()
.send(frame, len)
: Sends a raw Ethernet frame without the overhead ofbeginFrame()
/write()
/endFrame()
. This is similar toEthernetUDP::send(data, len)
.static constexpr int maxFrameLen()
: Returns the maximum frame length including the FCS. Subtract 4 to get the maximum length that can be sent or received using this API.static constexpr int minFrameLen()
: Returns the minimum frame length including the FCS. Subtract 4 to get the minimum length that can be sent or received using this API.
The MDNS
object provides an mDNS API.
begin(hostname)
: Starts the mDNS responder and uses the given hostname as the name.end()
: Stops the mDNS responder.addService(type, protocol, port)
: Adds a service. The protocol will be set to"_udp"
for anything other than"_tcp"
. The strings should have a"_"
prefix. Uses the hostname as the service name.addService(type, protocol, port, getTXTFunc)
: Adds a service and associated TXT records.removeService(type, protocol, port)
: Removes a service.static constexpr int maxServices()
: Returns the maximum number of supported services.
The DNSClient
class provides an interface to the DNS client.
setServer(index, ip)
: Sets a DNS server address.getServer(index)
: Gets a DNS server address.getHostByName(hostname, callback)
: Looks up a host by name and calls the callback when there's a result.getHostByName(hostname, ip, timeout)
: Looks up a host by name.static constexpr int maxServers()
: Returns the maximum number of DNS servers.
The util/PrintUtils.h
file declares some useful output functions. Note that
this file is included when QNEthernet.h
is included; there's no need to
include it separately.
The functions are in the qindesign::network::util
namespace, so if you've
already added using namespace qindesign::network;
to your code, they can be
called with util::writeMagic()
syntax. Otherwise, they need to be fully
qualified: qindesign::network::util::writeMagic()
.
-
writeFully(Print &, buf, size, breakf = nullptr)
: Attempts to completely write bytes to the givenPrint
object. The optionalbreakf
function is used as the stopping condition. It returns the number of bytes actually written. The return value will be less than the requested size only ifbreakf
returnedtrue
before all bytes were written. Note that a NULLbreakf
function is assumed to returnfalse
.For example, the
EthernetClient::writeFully()
functions use this and pass the "am I disconnected" condition as thebreakf
function. -
writeMagic(Print &, mac, breakf = nullptr)
: Writes the payload for a Magic packet to the givenPrint
object. This useswriteFully()
under the covers and passes along thebreakf
function as the stopping condition.
There is also a Print
decorator for stdio
output files, util::StdioPrint
(assuming the qindesign::network
namespace is in scope). It provides a Print
interface so that it is easy to print Printable
objects to stdout
or
stderr
without having to worry about buffering and the need to flush any
output before printing a Printable
directly to, say, Serial
.
The core library version of IPAddress
is missing a ==
operator that can
compare const IPAddress
values. In addition, it is completely missing a !=
operator. Provided in this library are these two operators. They are declared as
follows in the usual namespace:
bool operator==(const IPAddress &a, const IPAddress &b);
bool operator!=(const IPAddress &a, const IPAddress &b);
This library works with both PlatformIO and Arduino. To use it with Arduino, here are a few steps to follow:
- Change
#include <Ethernet.h>
to#include <QNEthernet.h>
. Note that this include already includes the header for EthernetUDP, so you can remove any#include <EthernetUdp.h>
. (i.e. There isn't a need to include the<QNEthernetUDP.h>
header.) - Just below that, add:
using namespace qindesign::network;
- In
setup()
, just after initializingSerial
, setstdPrint = &Serial
. This enables some lwIP output and alsoprintf
for your own code. - You likely don't want or need to set/choose your own MAC address, so just
call
Ethernet.begin()
with no arguments. This version uses DHCP. The three-argument version (IP, subnet mask, gateway) sets those parameters instead of using DHCP, but starts DHCP if they're all zero (INADDR_NONE
). If you really want to set your own MAC address, seesetMACAddress(mac)
or one of thebegin
functions that takes a MAC address parameter. - There is an
Ethernet.waitForLocalIP(timeout)
convenience function that can be used to wait for DHCP to supply an address. Try 10 seconds (10000 ms) and see if that works for you. Ethernet.hardwareStatus()
always returns zero.- Most other things should be the same.
Please see the examples for more things you can do with the API, including:
- Using listeners to watch for network changes, and
- Monitoring raw Ethernet frames.
I'll start with these statements:
- Don't use the
print
functions when writing data to connections. - Always check the
write
andprint
(andprintln
andprintf
) return values, retrying if necessary. - Data isn't necessarily sent immediately.
The write
and print
functions in the Print
API all return the number of
bytes actually written. This means that you must always check the return value,
retrying any missing bytes if you want all your data to get sent.
For example, the following code won't necessarily send all 250ร102 bytes. Buffers might get full. There might be retries. Etcetera.
void sendTestData(EthernetClient& client) {
for (int i = 0; i < 250; i++) {
// 102-byte string (println appends CRLF)
client.println("1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890");
}
}
The following modification will print a message every time the number of bytes actually written does not match the number of bytes sent to the function. You might find that the message prints one or more times.
void sendTestData(EthernetClient& client) {
for (int i = 0; i < 250; i++) {
// 102-byte string (println appends CRLF)
size_t written = client.println("1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890");
if (written != 102) {
// This is not an error!
Serial.println("Didn't write fully");
}
}
}
The solution is to utilize the raw write
functions and retry any bytes that
aren't sent. Let's create a writeFully
function and use that to send the data:
// Keep writing until all the bytes are sent or the connection
// is closed.
void writeFully(EthernetClient &client, const char *data, int len) {
// Don't use client.connected() as the "connected" check because
// that will return true if there's data available, and this loop
// does not check for data available or remove it if it's there.
while (len > 0 && client) {
size_t written = client.write(data, len);
len -= written;
data += written;
}
}
void sendTestData(EthernetClient& client) {
for (int i = 0; i < 250; i++) {
// 102-byte string
size_t written = writeFully(client, "1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"\r\n");
}
}
Note that the library implements writeFully
; you don't have to roll
your own.
Rewriting this to use the library function:
void sendTestData(EthernetClient& client) {
for (int i = 0; i < 250; i++) {
// 102-byte string
size_t written = client.writeFully("1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"1234567890"
"\r\n");
}
}
Let's go back to our original statement about not using the print
functions.
Their implementation is opaque and they sometimes make assumptions that the data
will be "written fully". For example, Teensyduino's current
print(const String &)
implementation attempts to send all the bytes and
returns the number of bytes sent, but it doesn't tell you which bytes were
sent. For the string "12345"
, print
might send "12"
, fail to send "3"
,
and successfully send "45"
, returning 4
.
Similarly, we have no idea what print(const Printable &obj)
does because the
Printable
implementation passed to it is beyond our control. For example,
Teensyduino's IPAddress::printTo(Print &)
implementation prints the address
without checking the return value of the print
calls.
Also, most examples I've seen that use any of the print
functions never check
the return values. Common practice seems to stem from this style of usage.
Network applications work a little differently, and there's no guarantee all the data gets sent.
The write
functions don't have this problem (unless, of course, there's a
faulty implementation). They attempt to send bytes and return the number of
bytes actually sent.
In summary, my strong suggestion is to use the write
functions when
sending network data, checking the return values and acting on them. Or you can
use the library's writeFully
functions.
See the discussion at: https://forum.pjrc.com/threads/68389-NativeEthernet-stalling-with-dropped-packets
Data isn't necessarily completely sent across the wire after write
or
writeFully
calls. Instead, data is merely enqueued until the internal buffer
is full or a timer expires. Now, if the data to send is larger than the internal
TCP buffer then data will be sent and the extra data will be enqueued. In other
words, data is only sent when either the buffer is full or an internal timer
has expired.
To send any buffered data, call flush()
.
To quote lwIP's tcp_write()
docs:
Write data for sending (but does not send it immediately).
It waits in the expectation of more data being sent soon (as it can send them more efficiently by combining them together). To prompt the system to send data now, call tcp_output() after calling tcp_write().
flush()
is what always calls tcp_output()
internally. The write
and
writeFully
functions only call this when the buffer is full. The suggestion is
to call flush()
when done sending a "packet" of data, for some definition of
"packet" specific to your application. For example, after sending a web page to
a client or after a chunk of data is ready for the server to process.
The examples aren't meant to be simple. They're meant to be functional. There are plenty of Arduino-style Ethernet library examples out there. This library does not aim to be just like the others, it aims to provide some more powerful tools to enable more powerful programs. The Teensy 4.1 is a serious device; it demands serious examples.
An attempt was made to use a more robust programming style and modern C++ tools. For those not as experienced with C++ or larger projects, these may have a steeper learning curve. Hopefully this brings up lots of questions and an exposure to new concepts. For example, state machines, lambdas, callbacks, data structures, vector::emplace_back, and std::move.
For those that do have more C++ and larger project experience, I invite you to improve or add to the examples so that the set of examples here becomes what you've always hoped Teensy library examples could be.
Hopefully this disambiguates some details about what each function does:
connected()
: Returns whether connected OR data is still available (or both).operator bool()
: Returns whether connected (at least in QNEthernet).available()
: Returns the amount of data available, whether the connection is closed or not.read
: Reads data if there's data available, whether the connection's closed or not.
Connections will be closed automatically if the client shuts down a connection,
and QNEthernet will properly handle the state such that the API behaves as
expected. In addition, if a client closes a connection, any buffered data will
still be available via the client API. If it were up to me, I'd have swapped the
meaning of operator bool()
and connected()
, but see the above list as
a guide.
Some options:
- Keep checking
connected()
(oroperator bool()
) andavailable()
/read
to keep reading data. The data will run out when the connection is closed and after all the buffers are empty. The calls toconnected()
(oroperator bool()
) will indicate connection status (plus data available in the case ofconnected()
or just connection state in the case ofoperator bool()
). - Same as the above, but without one of the two connection-status calls
(
connected()
oroperator bool()
). The data will just run out after connection-closed and after the buffers are empty.
There are a few ways in the API to utilize multicast to send or receive packets. For reception, you must join a multicast group.
The first is by using the EthernetUDP::beginMulticast(ip, port)
function. This
both binds to a specific port, for only receiving traffic on that port, and
joins the specified group address. It's similar to
EthernetUDP::begin(localPort)
in that the socket will "own" the port.
Since only one socket at a time can be bound to a specific port, you will need
to use the same socket if you want to receive traffic sent to multiple groups on
the same port. You can accomplish this by either calling
beginMulticast(ip, port)
multiple times, or by using begin(localPort)
once,
and then calling Ethernet.joinGroup(ip)
for each group you want to join.
To send multicast traffic, simply send to the appropriate IP address. There's no need to join a group.
The lwIP stack keeps track of a group "use count". This means:
- That
joinGroup(ip)
can be called multiple times, it just needs to be paired with a matching number of calls toleaveGroup(ip)
. Each call increments an internal count. - Each call to
leaveGroup(ip)
decrements a count, and when that count reaches zero, the stack actually leaves the group.
It's possible to register mDNS services. Some notes:
- Similar to
Ethernet
, there is a globalMDNS
object. It too is in theqindesign::network
namespace. - It's possible to add TXT items when adding a service. For example, the
following code adds "path=/" to the TXT of an HTTP service:
You can add more than one item to the TXT record by adding to the vector.
MDNS.begin("Device Name"); MDNS.addService("_http", "_tcp", 80, []() { return std::vector<String>{"path=/"}; });
- When adding a service, the function that returns TXT items defaults to NULL,
so it's not necessary to specify that parameter. For example:
MDNS.begin("Device Name"); MDNS.addService("_http", "_tcp", 80);
The library interfaces with DNS using the DNSClient
class. Note that all the
functions are static.
Things you can do:
- Look up an IP address by name, and
- Set multiple DNS servers.
The Ethernet.setDNSServerIP()
function sets the zeroth DNS server.
Internally, lwIP uses printf
for debug output and assertions. The library
defines _write()
so that printf
will work for stdout
and stderr
. It
sends output to a custom variable, Print *stdPrint
, that defaults to NULL. To
enable any lwIP output, including assertion failure output, that variable must
be set to something conforming to the Print
interface. If it is not set,
printf
will still work, but there will be no output.
For example:
void setup() {
Serial.begin(115200);
while (!Serial && millis() < 4000) {
// Wait for Serial initialization
}
qindesign::network::stdPrint = &Serial;
}
The side benefit is that user code can use printf
too.
If your application wants to define its own _write()
implementation, then the
internal one needs to be declared as weak. To accomplish this, define the
QNETHERNET_WEAK_WRITE
macro.
There is support for sending and receiving raw Ethernet frames. See the
EthernetFrame
API, above.
The lwIP stack supports VLAN tagging. Here are the steps for how to implement
it. Note that all defines should go inside lwipopts.h
. Documentation for these
defines can be found in src/lwip/opt.h.
- Define
ETHARP_SUPPORT_VLAN
as1
. - To set VLAN tags, define
LWIP_HOOK_VLAN_SET
. - To validate VLAN tags on input, define one of:
LWIP_HOOK_VLAN_CHECK
, (seeLWIP_HOOK_VLAN_CHECK
)ETHARP_VLAN_CHECK_FN
, (seeETHARP_SUPPORT_VLAN
)ETHARP_VLAN_CHECK
. (seeETHARP_SUPPORT_VLAN
)
I'm not 100% percent certain where this library will go, but I want it to be amazing. It's meant to be an alternative to the NativeEthernet/FNET library, using a different underlying TCP/IP stack.
I'm also not settled on the name.
Input is welcome.
- Tune lwIP.
- A better API design than the Arduino-defined API.
- Perhaps zero-copy is an option.
- Make a test suite.
- I have seen
Assertion "tcp_slowtmr: TIME-WAIT pcb->state == TIME-WAIT" failed at line 1442 in src/lwip/tcp.c
when sending a large amount of data. Either it's an lwIP bug or I'm doing something wrong. See: https://lists.gnu.org/archive/html/lwip-users/2010-02/msg00013.html - More examples.
- IPv6.
- IEEE 1588 features.
Code style for this project mostly follows the Google C++ Style Guide.
Other conventions are adopted from Bjarne Stroustrup's and Herb Sutter's C++ Core Guidelines.
- lwIP testing by manitou: https://forum.pjrc.com/threads/60532-Teensy-4-1-Beta-Test?p=237096&viewfull=1#post237096
- Dan Drown's NTP server and 1588 timestamps: https://forum.pjrc.com/threads/61581-Teensy-4-1-NTP-server
- Paul Stoffregen's original Teensy 4.1 Ethernet code: https://github.com/PaulStoffregen/teensy41_ethernet
- Dan Drown's modifications to Paul's code: https://github.com/ddrown/teensy41_ethernet
- Tino Hernandez's (vjmuzik) FNET-based NativeEthernet library: https://forum.pjrc.com/threads/60857-T4-1-Ethernet-Library
- Arduino Ethernet library
Copyright (c) 2021-2022 Shawn Silverman