w3c/respec-web-services

ReSpec's website and RESTful APIs

respec.org

ReSpec's website and RESTful API

https://respec.org

respec-web-services's Issues

xref: optimize storage with aliases

See sidvishnoi/respec-xref-route#54

Webref not longer using "master"

WebRef switched to using "main" as its primary branch, so service not longer updating.

The standard xref UI cannot search for terms with `.` in them

PaymentAddress.postalCode shows up as a suggestion in xref, thus implying that xref has its its definition. @sidvishnoi confirms that xref has the correct definition.

Expected:

1. Searching for PaymentAddress.postalCode in the Terms, concept, or WebIDL field on xref locates the correct spec and shows the result.

Actual:

1. Searching for PaymentAddress.postalCode in the Terms, concept, or WebIDL field on xref does not locate any specs and shows No results found.
2. To correctly locate the spec, one has to type postalCode in the Terms, concept, or WebIDL field, PaymentAddress in the for field and check the Advanced Search box.

Reason and Suggested Fix

@sidvishnoi confirms that this is because the standard search UI only finds terms without .. Ideally, if a term shows up as suggested on xref, one should be able to click it and get the result as normal.

Thank you for helping me debug this, Sid!

WebRTC WG Patent Policy out of date

https://respec.org/w3c/groups/webrtc says the WebRTC WG is under PP2017, but it was switched to PP2020 last Friday. The data returned by the W3C API is uptodate - not sure what needs to be put in motion to update the respec data.

Return "task-source" type in search

When searching, we should return return things marked as "task-source"

Same with permission

xref: IDL constants not exposed

e.g. HAVE_METADATA isn't exposed in xref, despite the definition in the HTML spec being correctly identified as public in webref; I havent' done a thorough check, but it looks like none of the IDL constants are exposed.

see also w3c/mediacapture-main#792 (comment)

Set up monitoring

https://www.w3.org/2020/12/express-monitor/monitor.html?services=https://labs.w3.org/normative-references/monitor;https://labs.w3.org/transition-issues-bot/monitor;https://labs.w3.org/github-cache/monitor;https://labs.w3.org/transition-notifier/monitor

SyntaxError object missing from ECMAScript data?

I need to check this in more detail, but see:
https://respec.org/xref/?term=TypeError

And:
https://respec.org/xref/?term=SyntaxError

DOMExceptions can be a "SyntaxError", but ECMAScript define {{ SyntaxError }} as an object too.

Check for data dir

npm run fails crashes if data dir is missing. We should create it if not there.

caniuse: group browser versions with no change

See sidvishnoi/respec-caniuse-route#8

Create a GitHub action for healthcheck

Create a GitHub action to run every 5 min or so to check the status of respec.org and send a message on Slack if server is down. Earlier, we were using webtask.io, but it's deprecated.

Code from webtask lambda function:

const fetch = require("node-fetch");
const { URL } = require("url");

const BASE = "https://respec.org/";

/**
 * @param context {WebtaskContext}
 */
module.exports = async (context, cb) => {
  const urls = ["/xref/meta/version", "/xref"];
  const results = await Promise.all(urls.map(test));
  if (results.some(result => !result.ok)) {
    await postToSlack(context.secrets.slack_webhook_url, results);
    cb(null, 'not cool');
  } else {
    cb(null, 'ok');
  }
};

async function test(pathname) {
  const url = new URL(pathname, BASE);
  url.searchParams.append("healthcheck", true);
  const { ok, status } = await fetch(url, { method: "HEAD" });
  return { pathname, ok, status, };
}

async function postToSlack(url, results) {
  const body = {
    text: results.map(e => `${e.ok ? '🟢' : '🔴'} HEAD ${e.status} ${new URL(e.pathname, BASE)}`).join("\n")
  };
  await fetch(url, {
    method: "POST",
    headers: {
      "Content-Type": "application/json"
    },
    body: JSON.stringify(body)
  });
}

Cannot read property 'includes' of undefined

Looks like req.headers can sometimes be missing at:

https://github.com/marcoscaceres/respec.org/blob/ee1252a4b0e7722b5736210f5859349f1282f84d/routes/w3c/group.js#L43

TypeError: Cannot read property 'includes' of undefined
    at route (/home/dh_ie5g4j/respec.org/routes/w3c/group.js:43:28)
    at Layer.handle [as handle_request] (/home/dh_ie5g4j/respec.org/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/dh_ie5g4j/respec.org/node_modules/express/lib/router/route.js:137:13)
    at cors (/home/dh_ie5g4j/respec.org/node_modules/cors/lib/index.js:188:7)
    at /home/dh_ie5g4j/respec.org/node_modules/cors/lib/index.js:224:17
    at originCallback (/home/dh_ie5g4j/respec.org/node_modules/cors/lib/index.js:214:15)
    at /home/dh_ie5g4j/respec.org/node_modules/cors/lib/index.js:219:13
    at optionsCallback (/home/dh_ie5g4j/respec.org/node_modules/cors/lib/index.js:199:9)
    at corsMiddleware (/home/dh_ie5g4j/respec.org/node_modules/cors/lib/index.js:204:7)
    at Layer.handle [as handle_request] (/home/dh_ie5g4j/respec.org/node_modules/express/lib/router/layer.js:95:5)

xref: add exceptions for some terms from HTML

Terms like link type "manifest" are used often, but they're not exported (Shepherd data allowed them, but reffy correctly doesn't). This cases some xref errors. We should add a way to support some exceptions to the current scraper rules.

w3c/groups: add and mark community groups

Need to add all community groups to w3c/groups.json and also add a distinction between WGs and CGs in API response so it can be used for w3c/respec#3017

Reverted health check

Was failing:
https://github.com/marcoscaceres/respec.org/actions/runs/103077881

So was getting emails every 5 mins

expose xref etc via w3c labs

Support for TAG and AB in conf.group

Is your feature request related to a problem? Please describe.

TAG and AB are authorized to publish in /TR.

Describe the solution you'd like

One should be able to specify one of
group: "ab"
group: "other/ab"
group: "tag"
group: "other/tag"

For that, the following links would need to work:

https://respec.org/w3c/groups/ab
https://respec.org/w3c/groups/ab/other
https://respec.org/w3c/groups/tag
https://respec.org/w3c/groups/tag/other

Note that
https://respec.org/w3c/groups/tag
but returns the wrong type (should be 'other', and not 'misc')

Describe alternatives you've considered

Not supporting the following forms could be considered:
group: "other/ab"
group: "other/tag"
https://respec.org/w3c/groups/ab/other
https://respec.org/w3c/groups/tag/other

but it would be impossible to disambiguate if a CG adopts 'tag' or 'ab' for their shortname.

Can you or your organization fund the work or help with development?
Sure

Additional context
Process 2021 deployment.

xref: Some page visibility entries not showing up

The following entry for "external now visible algorithm" is not showing up in xref for some reason:

https://github.com/w3c/webref/blob/cd238a8986d9c1355ff200e29db21f896c72d43f/tr/dfns/page-visibility-2.json#L228-L244

how to site shouldn't have spaces

The "how to cite" shouldn't have spaces for the micro syntax... as it makes a mess when tidy is run.

Add logging

Maybe use morgan

cache CORS preflight requests

We can cache the preflight requests with cors({ maxAge }).

New groups

Here are some active CGs that might be useful to add to w3c/groups:

• GPU for the Web Community Group
• ACT Rules Community Group
• XForms Users Community Group
• MathML Refresh Community Group
• MiniApps Ecosystem Community Group

We should also figure out the name convention for the WG-CG groups like:

• Immersive Web WG / Immersive Web CG
• WebAssembly WG / WebAssembly CG
• Second Screen WG / Second Screen CG

xref: duplicate entries in xref

See sidvishnoi/respec-xref-route#81

Make it easy to get started

Running on a new device is overwhelming. We should:

• Fix #123 and rest of mkdir -p related issues
• make update-data-sources work properly (no need to run w3c update script separately)
• read .env file in CLI scripts also
• create /docs/ on first start also

Weighted trie

We should look at using a weighted trie for the searches based on most commonly searched for...

As a bonus, we could show the most popular searches :) .... or the "term of the day" of the day 😹

Use top-level await where possible

When I update my github issues (label for example) they are not updated in the respec, even if I reload the page

Important info

• URL to affected spec: https://dcat-ap-ch-def.clients.liip.ch/releases/2.0/dcat-ap-ch.html
• ReSpec version: 26.8.14
• github repo: https://github.com/opendata-swiss/dcat_ap_ch

Problem:

My issue paragraphs pull in old labels and text instead of the updated labels and content, when I update my issues on github, see here https://dcat-ap-ch-def.clients.liip.ch/releases/2.0/dcat-ap-ch.html#issue-summary
Issue Nr,19 gets the wrong label. Can that be changed?

Expose information on patent policy under which a WG operates

In my work towards facilitating transition to Process 2020 in w3c/respec#3086, I'm proposing to add a new configuration option wgPatentPolicy that determines whether a group is operating under Patent Policy 2017 or the newly approved Patent Policy 2020.

Ideally, that information would be determined automatically for specs where the group info is retrieved from the W3C API. It is exposed in the route https://api.w3.org/groups/<id>/charters (when embed is set to true) on the last charter, as a property named "patent-policy".

So it would be great if the respec.org cache could expose that value so that the configuration option in ReSpec be determined automatically.

Block robots

We should add robot.txt:

User-agent: * 
Disallow: /

Expose attr-value types and how to cite in /xref

For example, <link rel=manifest> is exported from html, and should be citable as [^link/rel/manifest^].

Similarly, we want:

[^html-global/role/button^]

Allow showing all the terms for a spec

When searching for a particular spec (s), but no term is provided:

We should return all the terms of that spec. The use case is to see why a term can't be found or matched... usually because it's spelled differently or is using a synonym.

xref: include dfns from TR documents also

Part of w3c/respec#3246

Allow empty searches (when other things are given)

It would be nice to allow, for instance:

"show me the enum values for the X spec" without needing to put in a search term.

Likewise, it would be nice to retrieve by type... e.g., "show me 'permissions' for all specs"

Wrong patent link for i18n WG

Important info

• URL to affected spec: https://w3c.github.io/clreq/
• ReSpec version: 26.13.4
• I did a "hard refresh", but it's still busted.

Description of problem

What happened (e.g., it crashed)?:

The i18n WG is chartered under the Patent Policy 2020, but ReSpec generates the Patent Policy 2017 link, which blocks the publication of the spec:

This document was produced by a group operating under the 1 August 2017 W3C Patent Policy.

Expected behavior (e.g., it shouldn't crash):

It should use PP 2020.

Optional, steps to reproduce:

1. Open https://github.com/w3c/clreq/blob/gh-pages/index.html locally (or open https://w3c.github.io/clreq/)
2. Go to "Status of This Document"

Should I add the wgPatentPolicy configuration option to the spec?

Use shortname from W3C API

To avoid bugs such as the one fixed in #113, it would be best to get the shortname of the groups from the W3C API to populate routes/w3c/groups.json - they are now exposed on the /group view (with key .shortname) thanks to @deniak

Remove deprecated res.redirect

express deprecated res.redirect(url, status). Use res.redirect(status, url).

https://github.com/marcoscaceres/respec.org/blob/65bd3d33ac7866494b58b93794b5d5495b0f565c/routes/w3c/group.js#L50

Add HSTS

Need to add to express, so automatically upgrades connection to tls

logo too big on mobile

We should make the logo a bit smaller:

caniuse: pre-process multi-version data

Caniuse data contains entries like 11.3-11.4 which when grouped with same-support entries appear like 11.3-11.4-13.4-13.7. We should split 11.3-11.4 entries into 11.3, 11.4 in the scraper, so the end-result becomes:

- 11.3-11.4-13.4-13.7
+ 11.3-13.7

Cannot read property 'padStart' of undefined

TypeError: Cannot read property 'padStart' of undefined
    at formatter (/home/dh_ie5g4j/respec.org/utils/logging.js:76:27)
    at Array.logRequest (/home/dh_ie5g4j/respec.org/node_modules/morgan/index.js:122:18)
    at listener (/home/dh_ie5g4j/respec.org/node_modules/on-finished/index.js:169:15)
    at onFinish (/home/dh_ie5g4j/respec.org/node_modules/on-finished/index.js:100:5)
    at callback (/home/dh_ie5g4j/respec.org/node_modules/ee-first/index.js:55:10)
    at Socket.onevent (/home/dh_ie5g4j/respec.org/node_modules/ee-first/index.js:93:5)
    at Socket.emit (events.js:327:22)
    at TCP. (net.js:670:12)

At:

  const unknown = chalk.dim.gray("-");

  return [
    chalk.gray(date),
>>  chalk.gray(remoteAddr.padStart(15)),
    request,
    referrer ? formattedReferrer : unknown,
    contentLength ? chalk.cyan(prettyBytes(contentLength)) : unknown,
    chalk.cyan(responseTime + " ms"),

Include or link to developer's guide

@msporny pointed out that the docs were missing a link (or the actual) developer guide...

We can probably just include it directly.

Invalid status code: undefined

RangeError [ERR_HTTP_INVALID_STATUS_CODE]: Invalid status code: undefined
    at ServerResponse.writeHead (_http_server.js:249:11)
    at ServerResponse.writeHead (/home/dh_ie5g4j/respec.org/node_modules/on-headers/index.js:44:26)
    at ServerResponse.writeHead (/home/dh_ie5g4j/respec.org/node_modules/on-headers/index.js:44:26)
    at ServerResponse.writeHead (/home/dh_ie5g4j/respec.org/node_modules/on-headers/index.js:44:26)
    at ServerResponse._implicitHeader (_http_server.js:240:8)
    at ServerResponse.end (/home/dh_ie5g4j/respec.org/node_modules/compression/index.js:103:14)
    at ServerResponse.send (/home/dh_ie5g4j/respec.org/node_modules/express/lib/response.js:221:10)
    at route (/home/dh_ie5g4j/respec.org/routes/w3c/group.js:65:28)
    at runMicrotasks ()
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

at:

    res.json(groupInfo);
  } catch (error) {
    const { statusCode, message } = error;
    res.set("Content-Type", "text/plain");
>>  res.status(statusCode).send(message);
  }
};

Search doesn't work in Safari

Throws:

[Error] Unhandled Promise Rejection: TypeError: undefined is not an object (evaluating 'specs.length')
getFormData (script.js:91)
(anonymous function) (script.js:98)
asyncFunctionResume
handleSubmit (script.js:97)
(anonymous function) (script.js:183)

xref-ui: use specref data

ReSpec makes use of specref for spec URLs, while xref-ui uses data from webref, adding confusion.
The "spec id" field in search results should show the one available in specref, or specref should be updated to ensure consistency.

Ambiguous last updated date

the date format at the bottom of the page appears to be US? Should use something not ambiguous (or write out the month).

xref not matching plural terms

Description of problem

What happened (e.g., it crashed)?:

Given:

    <section id="abstract" data-cite="dom">
      <p>[=elements=] [=attributes=]</p>
    </section>

Expected behavior (e.g., it shouldn't crash):

Expected xref to work on those, as they are just concepts.

caniuse: improve support titles to better match caniuse.com

https://github.com/marcoscaceres/respec.org/blob/2cb8ea577c835637edc6644a0e4ae3f92058f403/routes/caniuse/lib/constants.ts#L20-L28

Also improve language for multi-status ('yd' / 'ad') titles.

Move autocomplete to server

A few issues with auto-complete on the client...

1. it downloads 500kb in terms
2. it do doesn't include the type information + other potentially useful things - so we can't style the autocomplete in useful ways (e.g., IDL things should be shown CodeByDefault wit the type and what spec a term declared in, etc. directly in the autocomplete).

We should allow the server to stream autocomplete results.

Prevent concurrent executions of bikeshed update

Add a promise queue, which ensures the the exec calls in /xref/update are run in series of request, and not simultaneously (to prevent race conditions and data corruption and reduce server load)

Hide "advanced options"

The following are not very useful in practice:

They are only really useful for debugging purposes. We should hide them under "show advanced search" or something