w3c/wot-testing

Repository for WoT Testing purposes

Web of Things (WoT) Testing

General information about the Web of Things can be found on https://www.w3.org/WoT/.

There is also a WoT Plugfest/Testing wiki for a weekly web conference call.

---

Repository for Web of Things (WoT) Testing purposes.

• Events: Information related to events such as plugfests and testfests, including both logistics and results.
• Tools: Set of tools to assist with testing and generation of reports.
• Systems: Online test Things, systems, and examples.
• Data: Current information on implementations.

Previous Plugfest Project Results and Discussion

Plugfest project descriptions, results, and discussions are kept in issues. The following links reference the projects for current and past plugfests. Note that issues for past plugfests are closed but are not deleted and can still be referenced.

• 2022.03 Plugfest
• 2022.06 Testfest

wot-testing's Issues

Experiment with geospatial queries in directories

add simple query filter to directory service that takes a geospatial location (lat, long, opt elevation) and a radius (cylindrical) and (optional) height interval and filters TDs that fall into that region.

Add link type(s) for UI(s)

See map use case. Edit: also possibly relevant to Accessibility.

Reorganize List of Assertion CSV files

• Manual assertions should not be manually maintained, instead...
• Report generator tool should generate a file, "all.csv", that lists all assertions in the spec
• Each tool that does testing should have a file, say "tested.csv", that lists the ones that tool (for example, the AssertionTester in the playground) tests (note there can be more than one tool in the future...)
• The report generator should take the union of all "tested.csv" files as input, subtract it from all.csv, and generate manual.csv: the "leftover" assertions that need to be manually asserted

Mistake in the definition of security and securityDefinition in the context file should be detected

During the virtual F2F TD session on 2020-06-26 (see minutes), it was pointed out that TD spec errors like the one should be caught by the test.

Collect and Annotate TDs

We need a reasonably large set of validated TDs annotated with semantics (eg OneDM) for semantic query scenarios and testing.
These may or may not be associated with working systems, but if their status should be recorded.
These can go into "data/TDs" at the top level. Consider this a call for PRs...

Experiment with visualization in Maps

• drive geolocation requirements and test
• Google Maps, OpenStreet Maps

Plugfest outcome: Get Validation and Reporting tools ready before the plugfest

Would have been nice to have the reporting tools done before the plugfest so we can look at outcomes

Set up logistics section for 2020.09.Online plugfest

Organize github site with framework for online plugfest w/ TPAC 2020

Update assertion tester

update to https://github.com/thingweb/thingweb-playground/tree/v1.1-testfest1 branch

Reorg existing material to make it more consistent

reorg material to remove redundancy and make filename conventions etc consistent (so events sort by date, etc)

Demos for 2021-03 Plugfest

• TUM (15m)
  • Wade (10m)
  • Testbench (5m)
• Hitachi (5m)
  • Node-RED Discovery Directory Integration (5m)

Deferred:

• Siemens
  • Edi{TD}or

Need TM-checking tool

A CLI tool like

    check-model {TD}.tm.jsonld output.csv >&2 errors.log

that pulls in the TMs referenced by the TD, and recursively checks them. This could be implemented by generating a custom JSON Schema from a TM, and should check as many assertions as possible in the spec. Output should be a CSV file that relates to all the named assertions in the spec.

Should check "type" and "extends" relation types in links.

Note: currently the edi{TD}or tool can do this, but it needs to be run manually (using a GUI), and tests only if the terms are used. Perhaps we could extend this to be able to run it on the command line.

Project (2021-03 Plugfest): Geolocation

Description:

• Define/refine geolocation information model
• Add geolocation information to TDs and Things
• Prototype application, eg. map

Participants:

• Michael McCool
• Daniel Peintner

Links:

• Geolocation information model
• Previous map project

Activities:

• Share link to geolocation information model
• Update Intel TDs in repo with geolocation information
• Figure out right way to do links (look at SDF for guidance and document)
• Create SDF capability for geolocation capability, submit to OneDM group
• Make examples actually consistent with geolocation information model spec
• Update geolocation information model proposal with description of geolocationOf relation type for links
• Deploy live Things that report (faked...) geolocation data via properties (dynamic case)
• Make sure TDs with geolocation info are loaded into directory service
• Update map test application to use geolocation data read from directory service
• Add extension to JSON Schema to support validation

How to support Python?

Let's document various approaches here. Here are some useful links discussed in the F2F meeting:

• https://github.com/agmangas/wot-py - Prototype native WoT API support in Python
• https://pypyjs.org/ - Python emulator in Node.js (would possibly allow running Python code on top of a Node.js instance also running a Node.js implementation of the WoT Scripting API)

Extend discovery support in Node-RED to support peer-to-peer (eg DNS) based discovery

Done, by @k-toumura

Document VLAN setup for SoftEther VPN

The configuration of the SoftEther VPN needs to be fully captured so it can be recreated.

Plugfest outcome: Balance Validation and Experimentation

Need to clarify the relationship and balance between testing and exploration of new concepts.

Experiment with geolocation visualization

Project (2021-03 Plugfest): WADE (WoT API Development Environment)

With @Fady222 we are experimenting on using WADE for the following:

• Basic Thing interactions made easy
  

• Forming mashups via the MAGE plugin
  

Results

• Multiple Fixes to improve usability and fetching of non HTTP TDs and fixing CORS errors for HTTPS
• First binary release (https://github.com/tum-esi/wade/releases/tag/1.0.1)

Testing experience from oneM2M - Issue 517 from w3c/wot

Coming from w3c/wot#517 on Jul 2, 2018

Tests

• Interoperability testing
• Conformance testing
• Field testing

Test structure

Configuration --> Scenario --> Test Case

Configurations

• Device to device
• Device to cloud
• Device via Gateway to cloud
• ...

Scenarios

• Discovery
• ~Pairing
• ...

Test Cases

One per primitive within scenario

Coverage

Specification might be to heavy to generate full coverage with test cases

Look at real products and sort them into "profiles"

Define test cases within profiles

Notes

• Tests can help developers before starting their own implementations
• Split specification into "features" from an architectural view to help structuring testing
• TTCN-3 helps
• Test cases also validate standard specification itself

Define Plugfest Scenarios for OneDM

• What are some use cases for OneDM/WoT?
• What are the requirements to support these use cases, eg. semantic query support in directories, etc?
• How and when do we annotate the TDs?
• How do we get the annotated TDs into the directory?
• Can we enhance tools to take advantage of annotations, eg. Node-RED integration?

Create Testing Plan

For each deliverable, need an implementation plan for CR transition. Following might be useful inputs:

• https://github.com/w3c/wot-testing/blob/main/data/testing/plan.md
• w3c/transitions#178
• https://w3c.github.io/wot-thing-description/testing/report.html

Testing plan can be included as a section of the Implementation Report. For each deliverable, should define:

• what is a "feature"
• functional tests
• coverage
• roles (producer/consumer etc)
• security testing (fuzz testing, etc)

Annotate TDs with geolocation information

To test spatial queries with directory services, need to annotate TDs with geolocation.
See w3c/wot-thing-description#941
Let's use this one (just for the purposes of the plugfest): w3c/wot-thing-description#941 (comment)
from schema.org

Catalogue Major Accomplishment from 2020.09-10 Plugfest

• NHK Hybridcast of coffee (incl. TUM/Siemens mashup)
• Node-red/discovery integration (incl. NHK mashup)
• Farm/TUM mashup: display sprinkler status
• Geospatial prototyping and map visualization

Project (2021-03 Plugfest): Conformance Testing via wot-testbench

With @MarcusWolfSchmidt we are testing https://github.com/tum-esi/testbench to send TD-conform requests to Things and to generate test reports/results accordingly.

The results will be used to think of how to document such results and how to show some metrics to the user

Plugfest outcome: Need to define appropriate DNS-SD service types

The directory used _wot._http service type and _directory subtype. This made it possible to browse for all WoT services in the VPN, directories and things alike in one go.

However, this made it hard to distinguish between directories and things when browsing only using the service type:

$ avahi-browse _wot._tcp --resolve
+ vpn_tun0 IPv4 Plugfest Local Thing Directory                _wot._tcp            local
+ vpn_tun0 IPv4 myRasPiLED                                    _wot._tcp            local
= vpn_tun0 IPv4 Plugfest Local Thing Directory                _wot._tcp            local
   hostname = [iot-linksmart-white.local]
   address = [192.168.30.132]
   port = [8081]
   txt = ["version=v1.0.0-beta.15" "td=/td"]
= vpn_tun0 IPv4 myRasPiLED                                    _wot._tcp            local
   hostname = [ktorpi.local]
   address = [192.168.30.135]
   port = [1880]
   txt = ["retrieve=/.well-known/wot-td"]

Browse using the subtype:

$ avahi-browse _directory._sub._wot._tcp --resolve
+   eth0 IPv4 Plugfest Local Thing Directory                         _wot._tcp            local
=   eth0 IPv4 Plugfest Local Thing Directory                         _wot._tcp            local
   hostname = [iot-linksmart-white.local]
   address = [192.168.30.132]
   port = [8081]
   txt = ["version=v1.0.0-beta.15" "td=/td"]

Project (2021-03 Plugfest): RIOT OS implementation

The purpose of this test is to figure out potential issues with the WoT TD server implementation in RIOT OS. IPv6 addresses are shared on the slack.

How to for flashing the ESP32:
The working branch is wot_coap.
wot_coap_light_switch contains some helper scripts to install the ESP32 toolchain and dependencies.

• Use install_esp_toolchain.sh to install the ESP toolchain.
• Use install_esp_dependencies.sh to install the ESP dependencies
• Move into wot_coap_generation directory
• Change the WiFi setting in the Makefile.
• Compile the app with ./compile_esp_app.sh
• Flash it with: ./flash_esp_app.sh
• You can connect to the RIOT shell with APPLICATION=wot_test BOARD=esp32-wroom-32 make term after it successfuly flashed.
• Get the IPv6 with ifconfig on the RIOT shell.

Findings:

• Fix for IPv6 in node-wot still has some issues.
• ESP32s don't get IPv6 addresses all the time. They need to get rebooted a couple of times before they get one.
• Over the night most of the ESP32s completly died and weren't accessible over CoAP or via ping. The ones who were accessible had a very high ping.
• ESP32s have a very unstable WiFi connection.
• TD response isn't always correct on the first try, but when repeating the request you get often the correct response.

Plugfest outcome: Xpath features useful for discovery

JSONPath is simpler (and lacks a formal specification, and implementations vary...), but XPath is more full-featured and better specified. It turns out some of the XPath features are useful and important for directory search... and we can depend on them, since they are actually required by a specification.

Project (2021-09 Plugfest): Geolocation - Intel

Description:

• Ontology for embedding geolocation data in TDs and in Thing
• Example TDs and services
• Geolocation filters in SPARQL
• See Intel TDs and Project

WoT marketing and some thoughts on the Plugfest

I created a PR in wot-marketing, because I think it is the correct repo for this.

Some points on the Plugfest from an outside perspective:
It is quite hard to get in touch with the W3C. I don't know if this is on purpose or just an side effect of the design of the W3C. So my question would be: Do you want to have more guests in the Plugfest or do you only want to invite certain people?

I also got the impression that the W3C is quite slow. All these processes make sense for defining a standard with different parties. But, if you want to interact with a more dynamic community, this will just be an obstacle. This is why I think reaching out to communities should be done outside of the W3C and their processes.

When I take a look into the libraries and tools, I get the impression that they are mainly maintained by students and PhD candidates. Only the nodeJS implementation looks like it is maintained by hand full of full time developer. From my perspective the contributing companies to the WoT need to invest in a small team in order to maintain a couple of core libraries. Additionally to this it is probably also a good idea to have dedicated people for interacting with other open source communities in order to reach more developer. I don't say this has to be forever, but for the start this is probably a good idea to get the ball rolling.

Create a template for organizing a plugfest

Template should include least the following:

• basic logistics
• each participant's inputs
• inputs (prework, data collection, etc) and outputs (documentation, reports, etc.)
• use cases, scenarios, interop testing, and demo planning

Create a "participation" README for people joining a plugfest

what they have to do, short form
including VPN info (@mjkoster)

Create Outcomes Presentation for 2021.03 Plugfest

• Create 10m summary pres - with backup slides for extra info
• Check into wot/PRESENTATIONS for 2021.03 F2F

Please also use the comments on this issue to note any major outcome, issues discovered, etc.
The Plugfest report out will be on the last day of the F2F but I'd like to get a draft of the presentation done by next week's main call.

Rename TD and TM suffixes to be consistent

• rename to use .td.jsonld and .tm.jsonld
• update README

I think this is done now, and will be proposing closing in the next meeting

Oct 24th Break out scenario - Issue 588 from w3c/wot

Coming from : w3c/wot#588
I'm very impressed today's Dev meetup Demo. Many things from different companies work together based on WoT specification.

On the other hand, from WoT beginners's point of view, I'm afraid, it might to tough to understand what is done, what is achieved....

What points do we promote at break out session on Oct.24th.
There are some layers of our achieved points, as my impression.

1. from one domain standard to cross domain standards

e.g.
Panasonic client service application can control Panasonic home appliances
products.
Panasonic exports TDs according to WoT.
Then Siemens client service application who never knows home appliance
standards can control Panasonic home appliance products.

TD is the declaration of device which includes

• Who I am
• What category I belong to
• What APIs I provide
• How to use APIs
• Supported communication protocols
• Supported security method

2. mixture of local network and global internet

This is achieved by introducing WoT proxy.

3. semantic match

A client would like to turn off all devices by simple one trigger.
TD provides semantics such as who provide "turn off" functionalities.

4. security negotiation

What kind of security technology the device provides is described in
TD.

This is basic information which makes device and clients negotiate
under well-defined security methodologies.

5. WoT simulator

Designing and implementing WoT total system, engineers do NOT have to
prepare complete devices as hardware.
Some of them can be prepared as software, that is, WoT simulator.
After system development, it is easy to substitute WoT simulator to
real hardware device just by changing url address from WoT simulator
to WoT of real device.
And Oracle also provides WoT device's event stream simulater.

These software can also work as WoT compliant checker.

6. Node WoT

Node WoT parses scripting API.

7. NodeRed

The open tool named Node Red provides graphical programming of WoT client
software.
Then the converter from TD to NodeRed's node is provided.

Then, how about create demo group such as "Siemens and Panasonic", "Oracle
and Intel" according to this scenario.
And show our demo according to scenario to audience.

Project (2021-03 Plugfest): Enable Thing Model support for the ediTDor

Description:

• Enable Thing Model support for the ediTDor

Participants:

• Felix Paulini
• Stefan Pfahler

Links:

• https://github.com/eclipse/editdor

Activities:

• Detect that a JSON is a Thing Model and validate with JSONSchema
• Update popups for creating Things to support Thing Models
• Support the concept of placeholders by implementing a mapping feature

Plugfest outcome: ids needed for primary keys

Directories (and any database) need stable primary keys. Since ids are optional in TDs these cannot be used for that purpose. So we need a workaround, perhaps autogenerated ids specific to the database in a wrapper object.

Experiment with TDD Thing Description

In the latest draft of the Discovery document, we defined a Thing Description for a Thing Description Directory. I think we should try to consume it with node-wot and interact with it. So that we can test both implementations and see if there are some gaps. If no one already did it and if we'll have a live TDD I can volunteer and report the result here otherwise feel free to close this issue.

Project (2021-03 Plugfest): Create several TMs

Description:

• create different TM instances based on the new features provided in latest TD draft
• test the new Edit{TD}or tool to create valid TM (also see issue #115)

Participants:

• @sebastiankb and @fexpal

Activities:

• created TMs can be found here
• outcomes so far documented in Slack

Project (2021-03): Create OneDM/SDF Semantic Annotation Examples

Description:

• Update examples (don't have to be live devices) that use semantic annotations derived from SDF and OneDM.
• These would also be useful test cases for semantic search in directories using SPARQL.

Participants

• Michael Koster
• Michael McCool

To Do:

• Create examples that use SDF annotations (see MODBUS example)
• Propose updates to TD to allow objects for @type to allow contentType for URIs (see/comment on fragment identifier issues)
• Validate (if possible)
• Create PR to push into wot-testing repo
• Register examples with WoT TD Directory (can be sourced from wot-testing)
• Prototype some use cases (e.g. SPARQL (or JSON path?) query for all temp sensors)

Update github PR validation to validate TMs

Now have a github action to validate TDs, would be useful to extend to TMs.

Plugfest outcome: Usefulness of VPN

Essential for:

• Using mDNS hostnames (eg jv.local) to avoid issues with IPs changing due to DHCP
• Using local DNS-SD service for discovery of a directory

Useful:

• Testing NAT traversal using a proxy provided by one company and a device by another
• Local-only devices (no public internet interface)
• Devices with no security
• Proprietary devices that should not be made publically available

Future testing:

• Mapping from a fixed URL (eg on a printed QR code) to a directory service using a local hostname
• Other discovery mechanisms (eg DHCP)

Test cases needed for all OAuth2 flows

For all oauth flows in the 1.1 spec (client, code, device) test cases are needed as well as two implementations (to be safe, two each for producer and consumer). Let's collect the implementations and the test cases below and see if we can get full coverage (two tests per flow at least).

Known implementations:

• node-wot Device flow implementation issue - consumer/producer
• LinkSmart directory service - producer
• node-gen/Node-RED - consumer

Update Validation script to new version of Playground

Update link to playground to new version when it is ready and remove hacks in validation script, see PR
eclipse-thingweb/playground#125
and move to "tools" directory

Multiple Thing Directory Tracker Page - Issue 403 from w3c/wot

Coming from w3c/wot#403

Can be handled by tools or systems folders in this repository.

Annotate TDs with sensory modalities

In order to enhance accessibility when Things are used as part of an IoT user interface, e.g. an audible alert may have to substituted with a visible alert for a deaf person (or vice-versa for a blind person).

Experiment with time-limited queries in directories

add an experimental query type to directories to only return TDs that were updated/created in a certain time interval; could be based on times recorded by the directory instead of/in addition to data in the TDs

Move VPN instructions to "tools" directory and clean up

Project (2021-03 Plugfest): Node-RED integration

Description:

• Directory-based discovery and auto-population (in addition to existing mDNS discovery)
• (stretch goal) use CoRE-RD instance on IETF Hackathon
• (stretch goal) keyword-based filtering of results (using JSON Path queries, UI just has keywords to filter on)

Participant:

• Kunihiko Toumura (@k-toumura )

Links:

• Description in past Plugfest:
  • 2020.06,
  • 2020.09

Activities:

• Fix bugs in Node Generator, e.g. IPv6 address handling, CoAP binding, etc.
• Check availability of Things and Directories
• Create UI for querying Directory from Node-RED

Project (2021-09 Plugfest): Node-RED integration (SPARQL query)

Description:

• Directory-based discovery (using SPARQL) and auto-population

Participant:

• Kunihiko Toumura (@k-toumura)

Activities:

• Check availability of Things and Directories
• Create UI for querying Directory from Node-RED
• Test auto-population feature using participating Things

Current implementation uses following simple SPARQL query:

PREFIX td: <https://www.w3.org/2019/wot/td#>
PREFIX dc: <http://purl.org/dc/terms/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT DISTINCT ?id ?title ?desc
WHERE {
    ?id rdf:type td:Thing;
        dc:title ?title;
        dc:description ?desc.
    FILTER(contains(?desc, "${query}")).
}
LIMIT 100

(${query} = input from Node-RED search UI)