webbluetoothcg/manual-tests

Manual tests for verifying use of the Web Bluetooth API.

manual-tests's Issues

Create pull request template

Create a template for external pull requests. This should include guidelines for development and formatting - best practices which cannot be caught by automated tests.

Document test coverage status

Currently the Web Bluetooth manual test coverage is listed in the internal document. We can consider to move the test coverage in the internal document to https://github.com/WebBluetoothCG/manual-tests/blob/main/README.md or create a new one called ROADMAP.md or TEST_COVERAGE.md. This way the information is easier to discover when browsing the test code files.

For each row entry in the test coverage list, in addition to show status and the API like currently we have in the internal document.

Status	Function/Attribute
✅	BluetoothCharacteristicProperties.read

We can consider add another column to indicate that can be tested in what test directory in the repo or point to the test page (ex: https://webbluetoothcg.github.io/manual-tests/characteristic_read/)

Status	Function/Attribute	Test link
✅	BluetoothCharacteristicProperties.read	https://webbluetoothcg.github.io/manual-tests/characteristic_read/

Add manual tests for navigator.bluetooth.getDevices and BluetoothDevice.watchAdvertisements

Write test for implementation of https://webbluetoothcg.github.io/web-bluetooth/#dom-bluetooth-getdevices.

Automate Running Tests

Hi! My team at @bocoup is working on automating these bluetooth API tests for Chrome. We'd like to take a more general approach so other browsers can benefit from these tests once they have implemented the Web Bluetooth API. I want to outline our proposed approach here to get any preliminary feedback y'all might have.

Implementation overview

Browser agnostic implementation

• Implement a script for running the existing tests with tap.
  • Takes command line arguments to specify
    • the bluetooth device name to connect to (by either a string for exact match or a regex) and
    • the browser environment to test under.
  • This script will iterate through the tests in this repo, wrapping them in a basic tap test script that asserts the test passed.
  • Prints the debug information currently shown at the bottom of the test pages.
  • All tests will be run serially
• An interface that mocks necessary browser-specific implementations to complete these tests automatedly. Where possible we'll write browser non-specific implementations.
  • Utility to initialize the driver
  • Utility to create session/target
  • Utility to upload espruino code
  • Utility to run JS test code
  • Utility to teardown or reset session/target
  • Utility to shut down the driver

Chrome-specific driver implementation

After this, we will proceed to implement the mocked utilities functionality for Chrome. This will primarily entail implementing the above utilities with puppeteer.

Usage

$ cd run_tests/
$ npm install
$ npm run test -- --device [device name regex] --browser [browser name]

Folder structure

/run_tests/
  index.js (cli file)
  package.json (dependencies needed to run automated test suite)
  test.js (wrapper for existing html tests)
  driver-mock.js (interface mock)
  shared/
    shared.js (shared utilities not specific to any browser)
  chrome/
    driver.js (chrome implementations)
    ... (other chrome implementation files)

Questions for feedback

• Should we use TypeScript for the code written as part of this work? This adds some complexity, but is fairly small since currently no JavaScripst in the repo is run outside of a browser and there would be a clear division between TS and JS files in the folder structure.
• Should we use a different folder or path to contain the file structure outlined above? (for example /scripts/run_tests/ instead of /run_tests/)

Please let me know if there's anything I can further clarify!

Expose tests for running w/o need to clone repo

Publish all tests at a web site (likely https://pages.github.com/). This allows these tests to be easily run without the need to clone the repository and run a local web server.

Create BLE Throughput Rate Benchmark Manual Test for WebBluetooth

Current WebBluetooth manual test suite doesn't have benchmark tests for testing the throughput. This would be helpful to be used to validate WebBluetooth implementation and ensure consistent performance across different platforms.