w3c/webref

Machine-readable references of terms defined in web browser specifications

Webref

Description

This repository contains machine-readable references of CSS properties, definitions, IDL, and other useful terms that can be automatically extracted from web browser specifications (see a list of projects known to use the data). The contents of the repository are updated automatically every 6 hours (although note information about published /TR/ versions of specifications are updated only once per day).

Specifications covered by this repository are technical Web specifications that appear in browser-specs.

The main branch of this repository contains automatically-generated raw extracts from web browser specifications. These extracts come with no guarantee on validity or consistency. For instance, if a specification defines invalid IDL snippets or uses an unknown IDL type, the corresponding IDL extract in this repository will be invalid as well.

The curated branch contains curated extracts. Curated extracts are generated from raw extracts in the ed folder by applying manually-maintained patches to fix invalid content and provide validity and consistency guarantees. The curated branch is updated automatically whenever the main branch is updated, unless patches need to be modified (which requires manual intervention). Curated extracts are published under https://w3c.github.io/webref/ed/.

Additionally, subsets of the curated content get manually reviewed and published as NPM packages on a weekly basis:

• @webref/idl contains a curated version of the ed/idl folder.
• @webref/css contains a curated version of the ed/css folder.
• @webref/elements contains a curated version of the ed/elements folder.
• @webref/events contains a curated version of the ed/events folder.

Important: The curated extracts only contain data for specifications that are in good standing (to keep the number of manually-maintained patches minimal and manageable). The NPM packages only contain curated extracts of specifications that are in good standing and that target web browsers.

Important: Unless you are ready to deal with invalid content, we strongly recommend that you process contents of the curated branch or NPM packages instead of raw content in the main branch.

Available extracts

This repository contains raw and curated information about nightly versions of Web specifications in the ed folder, as well as raw information about the released version (for /TR/ specifications) in the tr folder.

Note: The tr folder only contains information about released specifications. Specifications that have not been published as /TR/ documents (such as WHATWG specifications or Community Group reports) do not appear under the tr folder in particular.

More often than not, released versions of specifications are much older than their nightly version. Data in the tr folder is more invalid/inconsistent than data in the ed folder as a result. Additionally, no attempt is being made at curating data in the tr folder, use the tr folder at your own risk!

The following subfolders in the curated branch contain individual machine-readable JSON or text files generated from specifications:

• ed/css: CSS terms (properties, descriptors, value spaces). One file per specification series.
• ed/dfns: <dfn> terms, along with metadata such as linking text, access level, namespace. One file per specification.
• ed/elements: Markup elements defined, along with the interface that they implement. One file per specification.
• ed/headings: Section headings. One file per specification.
• ed/idl: Raw WebIDL index. One file per specification series.
• ed/idlnames: WebIDL definitions per referenceable IDL name. One file per IDL name.
• ed/idlnamesparsed: Parsed WebIDL structure of definitions in the idlnames folder. One file per IDL name.
• ed/idlparsed: Parsed WebIDL structure of definitions in the idl folder. One file per specification.
• ed/ids: Fragments defined in the specification. One file per specification.
• ed/links: Links to other documents, along with targeted fragments. One file per specification.
• ed/refs: Normative and informative references to other specifications. One file per specification.

Individual files are named after the shortname of the specification, or after the shortname of the specification series for CSS definitions and raw IDL files. Individual files are only created when needed, meaning when the specification actually includes relevant terms.

The ed/index.json file contains the index of specifications that have been crawled, and relative links to individual files that have been created.

This repository uses Reffy, a Web spec exploration tool, to crawl the specifications and generate the data. In particular, the data it contains is the result of running Reffy. The repository does not contain any more data.

Raw WebIDL extracts are used in web-platform-tests, please see their interfaces/README.md for details.

Curation guarantees

Data curation brings the following guarantees.

Web IDL extracts

• All IDL files can be parsed by the version of webidl2.js referenced in package.json.
• WebIDL2.validate passes with the exception of the "no-nointerfaceobject" rule about [LegacyNoInterfaceObject], which is in wide use.
• All types are defined by some specification.
• All extended attributes are defined by some specification.
• No duplicate top-level definitions or members.
• No missing or mismatched types in inheritance chains.
• No conflicts when applying mixins and partials.

CSS extracts

• All values in CSS files can be parsed by the version of CSSTree used in peerDependencies in package.json.
• No duplicate definitions of CSS properties provided that CSS extracts of delta specs are not taken into account (such extracts end with -n.json, where n is a level number).
• CSS extracts contain a base definition of all CSS properties that get extended by other CSS property definitions (those for which newValues is set).
• All entries in CSS files that do not extend a base definition link back to their actual definition in the spec. In other words, all entries under properties[], properties[].values[], selectors[], atrules[] and values[] have an href key that contains an absolute URL with fragment, except properties that that have a newValues key, at-rules that neither have a prose nor a value key, and definitions of a delta spec that completely override a definition in a previous level.

Elements extracts

• All Web IDL interfaces referenced by elements exist in Web IDL extracts.
• All elements link back to their definition in the spec.

Events extracts

• All events have a type attribute that match the name of the event
• All events have a interface attribute to describe the interface used by the Event. The Web IDL interface exists in the latest version of the @webref/idl package at the time the @webref/events package is released, and represents an actual interface (i.e. not a mixin).
• All events have a targets attribute with a non-empty list of target interfaces on which the event may fire. All Web IDL interfaces in the list exist in the latest version of the @webref/idl package at the time the @webref/events package is released, and represent an actual interface (i.e. not a mixin).
• The bubbles attribute is always set to a boolean value for target interfaces that belong to a bubbling tree (DOM, IndexedDB, Serial API, Web Bluetooth).
• The bubbles attribute is only set for target interfaces that belong to a bubbling tree.
• The bubblingPath attribute is only set for target interfaces on which the event bubbles.
• The targets attribute contains the top most interfaces in an inheritance chain, unless bubbling conditions differ. For instance, the list may contain { "target": "Element", "bubbles": true } but not also { "target": "HTMLElement", "bubbles": true } since HTMLElement inherits from Element.
• For target interfaces that belong to a bubbling tree, the targets attribute only contains the deepest interface in the bubbling tree on which the event may fire and bubble. For instance, the list may contain { "target": "HTMLElement", "bubbles": true }, but not also { "target": "Document" } since event would de facto fire at Document through bubbling.

Known consumers

The following projects are known to use webref data:

• MDN content management system, yari
• Web Platform Tests
• Open Web Docs MDN Browser Compat Data collector
• bikeshed, a specification authoring tool
• ReSpec, a specification authoring tool
• "DOM" Bindings for TypeScript
• browser API bindings for Dart
• the Pulsar text editor
• PostCSS preset env plugin
• strudy
• nodysseus
• WebIDLPedia
• Webdex

Using webref data in a project that is not yet in the list? Let us know!

Potential spec anomalies

This repository used to contain analyses of potential spec anomalies, such as missing references and invalid Web IDL definitions. These analyses are now published in the companion w3c/webref-analysis repository.

How to suggest changes or report an error

Feel free to raise issues in this repository as needed. Note that most issues likely more directly apply to underlying tools:

• Errors in the data are most likely caused by bugs or missing features in Reffy, which is the tool that crawls and parses specifications under the hoods. If you spot an error, please report it in Reffy's issue tracker.
• If you believe that a spec is missing from the list, please check browser-specs and report it there.

Development notes

GitHub Actions workflows are used to automate most of the tasks in this repo.

Data update

• Update ED report - crawls the nightly version of specifications and updates the contents of the ed folder. Workflow runs every 6 hours. Specifications that have not been modified since last crawl are skipped, unless the version of Reffy changed in the meantime. A typical crawl takes a few minutes to complete. A full crawl takes up to 12mn.
• Update TR report - crawls the released version of specifications and updates the contents of the tr folder. Workflow runs once per day. A typical crawl takes a few minutes to complete. A full crawl takes up to 8mn.
• Curate data & Prepare package PRs - runs whenever crawled data gets updated and updates the curated branch accordingly (provided all tests pass). The job also creates pull requests to release new versions of NPM packages when needed. Each pull request details the diff that would be released, and bumps the package version in the relevant packages/xxx/package.json file.
• Clean up abandoned files - checks the contents of repository to detect orphan crawl files that are no longer targeted by the latest crawl's result and creates a PR to delete these files from the repository. Runs once per week on Wednesday. The crawl workflows does not delete these files automatically because crawl sometimes fails on a spec due to transient network or spec errors.
• Test - runs tests on pull requests.
• Clean patches when issues/PR are closed - drops patches that no longer need to apply because underlying issues got fixed. Runs once per week.

Releases to NPM

• Publish @webref/xx package if needed - publishes a new version of the @webref/css, @webref/elements, @webref/events or @webref/idl package to NPM, tags the corresponding commits on the main and curated branches, and updates the relevant @webref/xxx@latest tag to point to the right commit on the curated branch. Runs whenever a pre-release PR is merged. Note that the released version is the version that appeared in packages/css/package.json, packages/elements/package.json, packages/events/package.json or packages/idl/package.json before the pre-release PR is merged.
• @webref/xx release: Request review of pre-release PR - assigns reviewers to NPM package pull requests. Runs once per week.

webref's Issues

Let wpt-sync/sync.js dry run on PRs (currently will affect real wpt PRs)

Discovered on #23.

I will turn off Travis for PRs until this issue is fixed.

Too many valueParseError since Marc 16 2019 in w3c/crawl.json

Hey guys

Many thanks for publishing this!

Just notice that there are too many valueParseError since Marc 16 2019. They seem to be because of some encoding issue upstream.

Here's an example

          "<final-bg-layer>": {
            "value": "<‘background-color’> || <bg-image> || <bg-position> [ / <bg-size> ]? || <repeat-style> || <attachment> || <box> || <box>",
            "valueParseError": "Unrecognized token <‘background-color’>"
          },

https://github.com/tidoust/reffy-reports/blob/19a80a9de50cb62ad33f3987c63198d34bf17c15/w3c/crawl.json

aom.idl not updated (deleted) when IDL was removed from spec

In web-platform-tests/wpt#9759 (comment) @alice pointed out that https://wicg.github.io/aom/spec/ no longer has any IDL. This happened in WICG/aom@4a21b85#diff-f2d8594ce0e9581fcd475b98f2fe2a82, over a month ago.

The aom.idl files should be deleted, and it would be good with some process to do this automatically, or flag that human attention is required.

IDL delta spec not included in webref

https://github.com/w3c/browser-specs/blob/master/specs.json lists both https://www.w3.org/TR/css-transitions-1/ and https://drafts.csswg.org/css-transitions-2/, where the latter is a delta spec. It has some IDL:
https://drafts.csswg.org/css-transitions-2/#idl-index

However, that IDL is not included in webref.

Wrong link on repo page

The link at the top of https://github.com/tidoust/reffy-reports/ goes to https://w3c.github.io/reffy-reports/ instead of https://tidoust.github.io/reffy-reports/

Trigger wpt interfaces/ updates in master Travis job in this repo

Bonjour,

In https://github.com/foolip/wpt-idl-importer I've written half of a solution for web-platform-tests/wpt#7784, creating branches but not opening PRs.

That's triggered on a daily Travis cron job that runs slightly after when I've observed the updates in this repo to happen. However, it's still an extra delay and if updates to this repo start happening faster, wpt updates won't get any faster.

So, I think it would be a good idea to let a Travis job in this repo trigger the updates. I'd be happy to maintain it of course, and would probably write it in node to match Reffy itself.

Thoughts?

css3-conditional should be css-conditional

From web-platform-tests/wpt#11868 (comment)

Scrape should be from the non-levelled https://drafts.csswg.org/css-conditional/, and filename should stay as css-conditional.

cc @gsnedders

No IDL for dom.spec.whatwg.org

This seems kind of bad. We were hoping to use reffy-reports's crawl.json in a tool we're building to get a list of all globals defined by specs, but this is a pretty basic hole...

/cc @jackbsteinberg

"Cannot lock ref" error on Travis CI job

@foolip, latest Travis CI job correctly detected the update on webaudio.idl, correctly created the PR against WPT, and yet the job failed with (slightly reformatted to fit the screen):

{ Error: Command failed: git push fork reffy-reports/webaudio

To https://github.com/autofoolip/wpt.git

 ! [remote rejected] reffy-reports/webaudio ->
       reffy-reports/webaudio (cannot lock ref 'refs/heads/reffy-reports/webaudio': reference already exists)

error: failed to push some refs to 'https://github.com/autofoolip/wpt.git'
    at checkExecSyncError (child_process.js:629:11)
    at Object.execSync (child_process.js:666:13)
    at git (/home/travis/build/w3c/webref/wpt-sync/sync.js:141:22)
    at updatePullRequests (/home/travis/build/w3c/webref/wpt-sync/sync.js:186:13)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  status: 1,
  signal: null,
  output:
   [ null,
     '',
     'To https://github.com/autofoolip/wpt.git\n ! [remote rejected]
      reffy-reports/webaudio -> reffy-reports/webaudio
      (cannot lock ref \'refs/heads/reffy-reports/webaudio\': reference already exists)\n
      error: failed to push some refs to \'https://github.com/autofoolip/wpt.git\'\n' ],
  pid: 9155,
  stdout: '',
  stderr:
   'To https://github.com/autofoolip/wpt.git\n ! [remote rejected]
    reffy-reports/webaudio -> reffy-reports/webaudio
    (cannot lock ref \'refs/heads/reffy-reports/webaudio\': reference already exists)\n
    error: failed to push some refs to \'https://github.com/autofoolip/wpt.git\'\n' }

The command "if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then
  node wpt-sync/sync.js --wpt-dir wpt --build-url https://travis-ci.org/w3c/webref/builds/$TRAVIS_BUILD_ID
else
  node wpt-sync/sync.js --wpt-dir wpt --dry-run
fi" exited with 1.

Any obvious reason for such a failure?

Migrate reffy-reports repo to w3c organization

Tasks to perform:

• Find a way to redirect https://tidoust.github.io/reffy-reports/ URLs to new destination? There may be external projects that actually use the https://tidoust.github.io/reffy-reports/ URLs for now and it would be good to make sure that the migration is transparent for them. But is there a magic solution to create redirects for all existing files? Actually, is there a good solution to create redirects for non-HTML files?
  => done in tidoust/tidoust.github.io@25f8051
• Migrate or create target repo under w3c organization. If people would fancy a new name, now would be a good time...
  => new w3c/webref repo created
• Make sure that Travis CI and job actions are enabled on the target repo
• Update URL in .travis.yml
  => see #53
• Update source URL for commit message in sync.js
  => see #53
• Make sure that the target repo has a CONFIG_JSON secret... or simply update the GitHub workflow not to require that file anymore. Reffy no longer depends on the W3C API, so it no longer needs a W3C API key!
  => dropped in 993e7ef
• Create the right GH_TOKEN secret under Travis CI for the target repo as done for tidoust/reffy-reports for the auto generation of PRs to WPT (requires secret from @foolip as I don't think the old secret can be copied over)

Anything missing?

Cc @dontcallmedom, @foolip

Space in "readonly maplike<DOMString, object>" removed from WebRTC

https://w3c.github.io/webrtc-pc/#rtcstatsreport-object has this IDL:

[Exposed=Window]
interface RTCStatsReport {
    readonly maplike<DOMString, object>;
};

The source also has a space after "readonly":
https://github.com/w3c/webrtc-pc/blob/f3850347f2f0bcadbad716664cd1543eddd172ca/webrtc.html#L10086

But in reffy-reports, there's a space missing:
https://github.com/tidoust/reffy-reports/blob/497e8f76f4d342b2400c131670788fe04596e1fb/whatwg/idl/webrtc.idl#L591

Possibly this is a bug in some whitespace normalization?

Replace tabs with spaces

We occasionally hit a lint error:

ERROR:lint:interfaces/webdriver.idl:9: Tabs used for indentation (INDENT TABS)

Publishing @webref/idl not possible if only IDL has changed

Running npx lerna diff doesn't show the diff against the published package but against what's in Git, and the IDL files in packages/idl/ are not checked in. As a result, npx lerna publish refuses to do anything if only the IDL files have changed, which will be very common.

I haven't been able to find a workaround. Should we just check in the patched files in packages/idl/?

unidentified PermissionName in permissions.webidl

as the date and time of writing, webidl package under interface directory has permissions.idl,

it does use PermissionName as a type, according to documentation it is an enum, however it has not been extracted,

css-images lacks object-position from css-images-3

It seems it currently refers to css-images-4 but AFAICT it's an extension of Level 3. Maybe there should be css-images and css-images-4, just as there are css-grid and css-grid-3?

Deprecate whatwg vs w3c views of the world?

Given the recent announcement of the MoU between WHATWG and W3C, we could probably simplify some of the code and some of the reports by deprecating the WHATWG vs W3C views of the world.

(I wonder if the W3C TR view of the world is useful / used in any way)

Incorrect Travis CI Build URL?

The build URL that appears in PR created by sync.js seems incorrect. Last one was:
https://travis-ci.org/w3c/webref/builds/723107475
... but that build number does not exist. Right build number and right build URL should rather be:
https://travis-ci.com/github/w3c/webref/builds/182272553

I do not know why there exists a mismatch in the build numbers. However, there seems to be a TRAVIS_BUILD_WEB_URL environment variable. @foolip, any reason not to use that in .travis.yml instead of attempting to build the URL ourselves?

Publish CSS bits as @webref/css

As a follow-up to #63 we should also have a @webref/css package.

@saschanaz needs it for saschanaz/types-web#66, and I'm also still using webref directly in https://github.com/foolip/mdn-bcd-collector because I get CSS properties from it.

What should be in such a package? Is it simply the JSON files in ed/css/?

Release @webref/idl every Monday

Per w3c/browser-specs#252 (comment) it would be great to arrange to @webref/idl releases on Mondays.

@dontcallmedom @tidoust what would a process for this look like? If we have steps written down for how to do the release, could we take turns doing it, or even automate? Do you automatically publishing any other NPM packages?

wpt-sync/sync.js can revert recent manual changes to wpt/interfaces/*.idl

web-platform-tests/wpt#13307 was produced the the sync code in this repo, and looked like a trivial change. However, it was actually reverting manual updates to payment-request.idl in web-platform-tests/wpt#13303. Later, as the spec changes had propagated to reffy-reports, web-platform-tests/wpt#13328 was created to bring everything into sync again.

While manual changes are discouraged by the boilerplate, people will reasonably want to ignore that sometimes, especially when updating the implementation and tests together. Nothing serious happened this time because dictionaries aren't exercised well by idlharness.js, but if other IDL changes are reverted and are automatically imported into the repo the changes originated from and causes test failures, that would be very frustrating for everyone involved.

In web-platform-tests/wpt#13269 (comment) a similar situation occurred with speech-api.idl, but there I was aware of the problem and waited for things to settle first.

@lukebjerring FYI

Link to this project in the boilerplate of extracted IDL

To resolve web-platform-tests/wpt#11109, it would be great if the boilerplate included a reference to the tooling that did the extracting, namely this repo. In the README.md, we could then perhaps also mention its use in wpt, and link to wpt/interfaces/README.md, so that it's easy for everyone to work out how the changes trickle into wpt.

[wpt-sync] race condition between merging PR and creating a new one

web-platform-tests/wpt#22785 (comment) describes the sequence of events that caused web-platform-tests/wpt#22774 to first be created, and then right as it was merged web-platform-tests/wpt#22785 was created and later closed again due to there being no changes.

The failure mode is fairly harmless since no changes are lost, but confusing.

We can detect this situation (existing branch but no PR) and add something to the PR description when it happens, similar to what we do when the .idl file in WPT has recently been modified by someone other than @autofoolip.

@stephenmcgruer do you have other ideas on what would have helped?

PermissionName enum is not extracted

https://w3c.github.io/permissions/#permission-registry

For some reason, this enum PermissionName does not end up in the extracted permissions.idl.

Publish crawled data with fixes to NPM?

I just did an experiment in to package up webref/ed/idl/*.idl with fixes from a branch and publish it as https://www.npmjs.com/package/webidl-dfns.

IDL with fixes applied is needed for mdn-bcd-collector, and I wanted to have it as a regular dependency rather than pulling it from a branch that I keep changing.

However, having done that experiment, I wonder if there would be interest in publishing webref directly to NPM? In addition to include ed/ and tr/ as-is, we would also provide fixed versions of ed/idl/* which we guarantee will parse with a given version of webidl2.js.

One could go further if one wanted to with global consistency guarantees such as:

• No duplicate definitions
• No unknown types used
• No circular inheritance

mdn-bcd-collector already does some of this, but if it could be centralized, I think WPT would also be better off making use of released data with some guarantees.

Missing HTML heading defined thing

With the ol' Shepherd data, it was possible to reference, for example, [=link type "manifest"=] - which is defined in the HTML spec in a H5 (and exported).

As such, right now, I've got a bunch of broken links over in the Web Manifest spec that are trying to link to [=link type "manifest"=]. I'm wondering how we should deal with those?

• add support for headings as definitions in Webref?
• fix the HTML spec to add <dfn data-export="">?
• Something else?

Error in crawl.json reading property version-history of undefined

As of the time of this writing, many entries in https://tidoust.github.io/reffy-reports/whatwg/crawl.json contain no IDL. Instead they contain an error field:

TypeError: Cannot read property 'version-history' of undefined TypeError: Cannot read property 'version-history' of undefined\n    at fetch.then.then.s (/home/dom/github.com/reffy/src/lib/util.js:316:34)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:189:7)

Examples are entries 231 (worklets-1), 227 (webrc-stats), 225 (webmidi), and more.

Send an alert when the list of IDL files changes

Addition/Removal of IDL files to the reports is manual for now. This is mostly on purpose: we don't want to delete IDL files because of a transient network or other error that would make Reffy crash on a given spec.

Given that the list of IDL files crawled by Reffy is also maintained manually, the list of IDL changes is unlikely going to change without human intervention in any case. But then, that does happen, see #10.

Without necessarily changing the logic in place, it would be good to send an alert one way or the other when new IDL files are detected that are not yet tracked in reffy-reports or when the crawl fails to generate an IDL file that is being tracked in reffy-reports.

CSS package: Add test to guarantee the presence of a value definition

See discussion in #115 (comment)
Also see possible improvement to Reffy: w3c/reffy#538

keyboard-map spec is missing

Spec: https://wicg.github.io/keyboard-map/
Tests: https://wpt.fyi/results/keyboard-map

Ensure trailing newline in IDL files

In web-platform-tests/wpt#9837 one can see that there's a missing newline at the end of the file, and this seems to be the case for all of https://github.com/tidoust/reffy-reports/tree/master/whatwg/idl.

Including that missing newline will:

• stop git from complaining on command line and GitHub on web UI
• look right if you cat a file on the command line (which I do a lot)

pointerevents-extension is missing

Defined here:
https://w3c.github.io/pointerevents/extension.html

I noticed pointerevents-extension.idl and its tests while authoring web-platform-tests/wpt#11922

CSS package: consider additional parsing guarantees

See discussion in #115 (comment)

Index of IDL names

ed/index.json (and its equivalent for TR) give an entry point to most of the relevant files in webref, but that's not the case for idlnames and idnamesparsed. Maybe we should have an index.json in these subdirectories, or add an entry in the main index.json that can serve as an index for them?

Extract HTML/SVG tagname-interface mapping

Background: saschanaz/types-web#76

It would probably require some changes to the spec files.

Idle Detection is not indexed

If it were in WPT /interfaces/ we could use idl_test() rather than some custom wrapper.

cc @foolip

subresource-integrity missing?

Can't see a scrape of subresource-integrity IDL anywhere under whatwg.
Spec: https://w3c.github.io/webappsec-subresource-integrity/

Generate partial IDL fragments for CSS properties for WPT?

CSSOM defines up to three partial interfaces that are created for each supported property:

partial interface CSSStyleDeclaration {
  [CEReactions] attribute [LegacyNullToEmptyString] CSSOMString _camel_cased_attribute;
};
partial interface CSSStyleDeclaration {
  [CEReactions] attribute [LegacyNullToEmptyString] CSSOMString _webkit_cased_attribute;
};
partial interface CSSStyleDeclaration {
  [CEReactions] attribute [LegacyNullToEmptyString] CSSOMString _dashed_attribute;
};

It would be nice to have these tested in WPT, whether Reffy itself generates these or whether they're generated merely for the WPT export.

See also speced/bikeshed#549 for a Bikeshed feature request to add generation of these to Bikeshed, and w3c/csswg-drafts#5412 for potential pitfalls here (though as far as I'm aware no such property is currently defined)

Exclude mediaont-api.idl from at least whatwg/idl/

See web-platform-tests/wpt#11874 (comment).

This spec does use IDL, but I doubt very much that it will ever be implemented by any browser.

webrtc-identity spec is missing

Spec: https://www.w3.org/TR/webrtc-identity/

I see it appearing in crawled links, but it appears to be missing from the list of specs to crawl.

webstorage spec is missing

Spec: https://www.w3.org/TR/webstorage/
Tests: https://wpt.fyi/webstorage

Crawls fail at the setup phase

The last 4 crawls have failed at the Setup Reffy step with an "[error]No such file or directory" message. I cannot tell whether the error is during the creation of the config.json file or while running npm ci.

Cc @foolip FYI and in case you already bumped into such errors in the past.

Raw log:

2019-12-04T12:31:37.3649287Z ##[group]Run echo "***" | base64 --decode > config.json
2019-12-04T12:31:37.3650743Z �[36;1mecho "***" | base64 --decode > config.json�[0m
2019-12-04T12:31:37.3650888Z �[36;1mnpm ci�[0m
2019-12-04T12:31:37.3679235Z shell: /bin/bash -e {0}
2019-12-04T12:31:37.3679469Z ##[endgroup]
2019-12-04T12:31:37.3876971Z ##[error]No such file or directory
2019-12-04T12:31:37.3925685Z Cleaning up orphan processes

Corresponding bits in the raw log of last successful crawl:

2019-12-03T12:31:56.5566224Z ##[group]Run echo "***" | base64 --decode > config.json
2019-12-03T12:31:56.5566674Z �[36;1mecho "***" | base64 --decode > config.json�[0m
2019-12-03T12:31:56.5566777Z �[36;1mnpm ci�[0m
2019-12-03T12:31:56.5604591Z shell: /bin/bash -e {0}
2019-12-03T12:31:56.5604819Z ##[endgroup]
2019-12-03T12:32:04.8831917Z 
2019-12-03T12:32:04.8833968Z > [email protected] postinstall /home/runner/work/reffy-reports/reffy/node_modules/@snyk/cocoapods-lockfile-parser/node_modules/core-js
2019-12-03T12:32:04.8834778Z > node -e "try{require('./postinstall')}catch(e){}"
2019-12-03T12:32:04.8835124Z 
2019-12-03T12:32:04.9650180Z �[96mThank you for using core-js (�[94m https://github.com/zloirock/core-js �[96m) for polyfilling JavaScript standard library!�[0m
2019-12-03T12:32:04.9651288Z 
2019-12-03T12:32:04.9652214Z �[96mThe project needs your help! Please consider supporting of core-js on Open Collective or Patreon: �[0m
2019-12-03T12:32:04.9653018Z �[96m>�[94m https://opencollective.com/core-js �[0m
2019-12-03T12:32:04.9653962Z �[96m>�[94m https://www.patreon.com/zloirock �[0m
2019-12-03T12:32:04.9654435Z 
2019-12-03T12:32:04.9655275Z �[96mAlso, the author of core-js (�[94m https://github.com/zloirock �[96m) is looking for a good job -)�[0m
2019-12-03T12:32:04.9655632Z 
2019-12-03T12:32:05.0065580Z 
2019-12-03T12:32:05.0067371Z > [email protected] install /home/runner/work/reffy-reports/reffy/node_modules/puppeteer
2019-12-03T12:32:05.0067875Z > node install.js
2019-12-03T12:32:05.0068162Z 
2019-12-03T12:32:06.2998563Z 
2019-12-03T12:32:12.9652041Z Chromium downloaded to /home/runner/work/reffy-reports/reffy/node_modules/puppeteer/.local-chromium/linux-706915
2019-12-03T12:32:13.0073663Z 
2019-12-03T12:32:13.0075623Z > [email protected] postinstall /home/runner/work/reffy-reports/reffy/node_modules/core-js
2019-12-03T12:32:13.0076115Z > node postinstall || echo "ignore"
2019-12-03T12:32:13.0076371Z 
2019-12-03T12:32:13.1336269Z 
2019-12-03T12:32:13.1338156Z > [email protected] postinstall /home/runner/work/reffy-reports/reffy/node_modules/respec
2019-12-03T12:32:13.1338592Z > opencollective-postinstall
2019-12-03T12:32:13.1338693Z 
2019-12-03T12:32:13.2119277Z �[96m�[1mThank you for using respec!�[96m�[1m
2019-12-03T12:32:13.2131572Z �[0m�[96mIf you rely on this package, please consider supporting our open collective:�[22m�[39m
2019-12-03T12:32:13.2132644Z > �[94mhttps://opencollective.com/respec/donate�[0m
2019-12-03T12:32:13.2133055Z 
2019-12-03T12:32:13.3312167Z added 589 packages in 16.236s

Automatically delete files that have no matching specs in browser-specs

via a pull request

Trim any trailing whitespace

Some files, due to an inconsistent whitespace offset, end up with trailing whitespace.
e.g. https://github.com/tidoust/reffy-reports/blob/master/whatwg/idl/geolocation-API.idl

Note: when I was doing these scrapes, I treated each individual snippet independently when trimming any common leading whitespace, which had a nicer result for inconsistent snippets - not sure if that's trivial in your pipeline or not.

Daily update didn't happen this morning

I was monitoring for the changes in w3c/reffy@cadc777 to take effect this morning, as it seems previous commits by @dontcallmedom have been at a regular interval. Today the update didn't happen. Last commit was 39f5e00.

I can't tell if this is running as a Travis cron job or something else, so can't help diagnose any further that that :)

@tidoust @dontcallmedom

wpt PRs for new IDL files called "Update" instead of "Add"

web-platform-tests/wpt#13150 is an example of this. I tried to differentiate new files when writing the sync script so this is a bug.

WebIDL snippets with default values are being dropped

e.g. w3c/wake-lock@5c34f9a lead to https://github.com/web-platform-tests/wpt/pull/17904/files#diff-d83394caf586049a88f0c0cdd880c7bbL12

e.g. w3c/push-api@9179964 lead to https://github.com/web-platform-tests/wpt/pull/17826/files#diff-173f90f4b0be112b694720eda63bce08L11

Add Origin Policy to reffy

I'd like to add an idlharness.js test, and it seems like the first step is to get an auto-generated file into WPT's interfaces/ folder.

https://wicg.github.io/origin-policy/ is the spec in question.

SVGMarkerElement is missing from @webref/idl

@foolip, why did it need to be removed in #68? 👀

Check CSS/IDL extracts against dfns extracts

All terms defined in CSS and IDL extracts should have a corresponding definition in dfns extracts. Checking this invariant automatically would allow to detect cases where e.g. definitions of IDL terms are missing.

See additional context and discussion in w3c/reffy#419

Export WebIDL fragments

To help with https://github.com/GoogleChromeLabs/webidl-diff/blob/dataCollection/data/FetchSpecRunner-WebSpec-journal.js and imports into web-platform-tests

Ideally: one file per spec, whose name match the spec shortname