mdn / yari Goto Github PK
View Code? Open in Web Editor NEWThe platform code behind MDN Web Docs
License: Mozilla Public License 2.0
The platform code behind MDN Web Docs
License: Mozilla Public License 2.0
Currently we're using this technique in the SSR rendering:
const escapeDocumentJson = JSON.stringify(document).replace("</", "<\\/");
Another option is: https://github.com/mathiasbynens/jsesc
And yet another option is: https://www.npmjs.com/package/serialize-javascript
but our approach don't "code generate" JavaScript code in the HTML document. Either way, it might be worth investigating if there are ways to break out of the </script>
tag and cause an XSS.
Since stumptown-renderer is stateless and relatively dumb our best bet is to use meta tags for sending pre-fetching type of hints.
When rendering a page that will have <iframe src="https://someotherdomain/...
in it, we should inject something like <link rel="preconnect" href="https://someotherdomain/...
in the HTML.
Would it be worth it to break up the React work.
In the cli we can use React to stitch together the large HTML blobs and that'd make the, for example, video/index.html
and video/index.json
.
No need for hydration or any inlined react state to support hydration.
Then, on the client, we have a different React bundle that does different things. (And if done correctly, could be swapped out for Preact or svelte or whatever) It handles routing, XHR loading for routing, header stuff (e.g. autocomplete), etc. If routing triggers that the page has changed, it goes to XHR fetch the JSON for the new page and simply injects the whole thing as one large blob of dangerouslySetHTML
. We'd need to make that video.json
dumber. It could basically look like this:
{
"title": "<video> - The HTML video tag",
"body": "<h2>Overview</h2>\n<p>Bla bla bla...",
"sidebar": "<h3>References</h3>\n<ul>...",
"bcd": "html-video-bcd",
"some": "metadata",
}
That body
in the JSON would contain everything except the <div>
that displays the BCD table. Instead it could be a placeholder like <div id="bcd" data-id="html-video-bcd">
. Then, the client-side React bundle could do a second XHR request just to get the BCD data and hydrate that into this div.
Possible outcome:
Possible risks:
See https://github.com/vadimdemedes/generate-help as a possible candidate.
There are many advantages with using Docker.
In CI - Get out of however Travis does Node. This way the only determining input variable for what version of Node and yarn we get is the version string of node we specify in the Dockerfile
. For example, as of today Travis preinstalls an ancient version of yarn
and there's no roadmap to that updating.
If we build a docker image, that contains ALL the rendered out .html files ready to be served, we can simply ship that docker container from Travis to Dockerhub and use something like Netlify to run that docker image.
For local development, every human developer would get the exact same version of Node and Yarn.
For local development, with a good docker-compose you would no longer need to "Open two terminals and in one run this and in another run that". Just docker-compose up
and you get a complete dev environment.
In CI, we would potentially have better control of caching. Instead of the cryptic Travis way of caching files.
Docker is slow. Sometimes makes it hard to do hacky, but necessary, debugging within installed node_modules
.
Not everyone is as comfortable with Docker as they are with pure Node and stuff. Possibly more applicable to contributors but on the flip-side being able to just say "To get started, run docker-compose up
and you're off to the races".
Too soon? We're not entirely sure where we're going with the outcome of this project so perhaps it's fine to leave it a bit "pure" and flat at the moment.
Generally, we want something like 1h Cache-Control (public) for all HTML pages and for things like /static/css/2.02b14290.chunk.css
we want the max cache-control possible.
The /service-worker.js
URL needs to have max-age=0
for best practice.
The link here...
Plainly goes to https://github.com/mdn/browser-compat-data
It's the same on prod as of today: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video#Browser_compatibility
Why can't it go to https://github.com/mdn/browser-compat-data/blob/master/html/elements/video.json instead?
The data needed to build the table looks like this:
{
"html": {
"elements": {
"video": {
"__compat": { ...
So, to make the right URL you just need to take each key of the JSON until you dug down to "__compat" and join them with a /
and add .json
to the end and prefix the whole thing with https://github.com/mdn/browser-compat-data/blob/master/
.
At the moment, the main argument to the cli
is a list of files. E.g.
yarn run run ../stumptown/packaged/html/reference/elements/*.json
which is expanded by bash so it basically corresponds to typing
yarn run run ../stumptown/packaged/html/reference/elements/abbr.json
yarn run run ../stumptown/packaged/html/reference/elements/address.json
...
What would be ideal is to be able to pass a list of files or directories and if it's a directory I think it should drill down recursively by default. E.g.
yarn run run ../stumptown/packaged/html/reference ../stumptown/packaged/css/ some/other/directory some/other/specific/file.json
It needs to work in a way that if you don't put a SENTRY_DSN_PUBLIC
(or whatever it's called) in your .env
it should still work.
At the moment, when we server-side render a page (e.g. a document) we inject the JSON source (the JSON built from the Stumptown JSON) into the DOM as <script id="document" type="application/json>
. That way, when you do the first client-side render, it leverages this to avoid a XHR lookup. But 99% of that content is not needed for hydration.
The content of that script tag is about 30KB and 7.4KB gzipped. Since we mostly don't need it, we can avoid a render by stripping out almost all of it possibly. That would boost the initial load if the whole .html
file can be made smaller.
https://community.netlify.com/t/my-url-paths-are-forced-into-lowercase/1659 hopefully explains it well. You can see the effect here:
▶ curl -v https://stumptown.netlify.com/docs/Web/HTML/Element/video > /dev/null
...
< HTTP/2 301
< location: /docs/web/html/element/video/
...
One thing I've noticed when running the cli is that the errors thrown are hard to map to which document (aka. which stumptown .json) was the cause.
Perhaps I don't know how to use sourcemaps in Node. (remember the cli is babel'ified into dist/cli.js
) or perhaps it's just the nature of Node tracebacks being not great.
A quick fix would be to change, for example:
throw new Error(
`prose section '${name}' is not optional and not present in document.prose`
);
to something like:
throw new Error(
`prose section '${name}' is not optional and not present in document.prose (on ${document.title)`
);
...or something similar. That would make sense when using the cli and it might come in hand when we connect the cli with Travis. But it doesn't make sense to repeat the obvious when you're doing dev in the browser because which document it is is clear from the URL bar.
Another thing to ponder is that going into production; it might be nice to NOT throw errors and just use console.error()
instead. So whatever we do we could leverage process.env.NODE_ENV
.
▶ cat stumptown/packaged/html/elements/abbr.json | jq .html.elements.attributes
null
This is breaking make deployment-build
but it's not obvious where the bug lies. In Stumptown or in this repo's renderer business logic.
We don't have a search yet but I have a prototype for an autocomplete widget that can be used to navigate by typing in parts of the title. A flaw with this is that when the search input is small, there's no good way to decide which page to show first. For example "fo" should probably should probably present JavaScript/Reference/Global_Objects/Array/forEach ahead of Web/API/MediaKeyStatusMap/forEach because the latter is so much more popular.
After reading the tip Chris Mills recently learned with regards to setting role=img on an SVG, I did some more reading and found some additional things we can do to improve the a11y of our SVG images.
This means updating the current SVG elements of the icon system to make proper use of:
<title>
<description>
(as appropriate) More can be read here:
https://developer.paciellogroup.com/blog/2013/12/using-aria-enhance-svg-accessibility/
In the renderer, we include all the "state" necessary to render a page. It's encoded as JSON into the HTML file. That's so React can hydrate.
Also, if you click on an internal link, in the sidebar for example, it will only need to download the index.json
file which is a lot smaller.
That JSON is getting pretty big and I think there's room for improvement/optimizations.
I analyzed the sizes of sub-keys in the client/build/docs/Web/HTML/Element/**/*.json
files and concluded this:
related_content 5358 47%
prose 2405 21%
contributors 1627 14%
browser_compatibility 1307 11%
examples 412 3%
interactive_example_url 70 0%
mdn_url 58 0%
title 34 0%
attributes 2 0%
E.g. a json like this
This all gets gzipped and stuff but let's get performance as a primary thought now rather than an afterthought. But, let's also not prematurely optimize.
https://reach.tech/router is an alternative to react-router-dom made, I think, by the same guys.
Is this new router better? Skimming the docs it talks about doing accessibility "better" but not sure. Worth investigating what the difference is.
Hi, I'm colorblind and I have some difficulties reading the browser campatibility table at the end of each docs page. The issue I have is with distinguishing between "Full Support" and "Compatibility Unknown". To me, the two colors look the same. Also when I look at the legend at the bottom, I see no difference between the two colors picked for this. I see some difference between "No Support" and the others, but there the cross helps me best.
I would suggest to pick colors that are more widely apart from eachother in the color spectrum and to perhaps also introduce an icon of some sort (like the cross for No Support) for all statuses. This would help me out a lot.
Thanks and regards,
Martijn
很抱歉,我不知道是不是Chrome浏览器的原因,但是我在HTML板块使用Chrome浏览器的google翻译可以进行英文翻译中文,但是在JavaScript板块中却始终无法进行翻译,所以我觉得是JavaScript板块进行了某些限制,我想问一下,是不是这个原因?
PS:因为中文版mdn有许多过时,所以只好用google翻译看英文版mdn
[Translation via Google]
Sorry, I don't know if it's the reason for the Chrome browser, but I can translate the Chinese in English using the google translation of the Chrome browser in the HTML section, but I can't translate it in the JavaScript section, so I think it is the JavaScript section. Some restrictions, I want to ask, is this the reason?
PS: Because the Chinese version of mdn has a lot of obsolescence, so I have to use google translation to see the English version of mdn
In the CLI, you might, at least in make deployment-build
iterate over every single document. That gives you opportunities. For example, ...
sitemap.xml
since you will have encountered every possible URL./docs/Web/HTML
)The buildHtmlAndJson
function takes a (stumptown json file) file path as input and then it has its own side-effect. But it doesn't really collect what it's doing. So you can't, after all file paths have been consumed, see what it built.
This issue is about an API refactor so that, upon building a HTML page, you can access what other documents are available. And at the end of the whole loop, a possibility to get insight into all documents and which ones were built. Enough to be able to build something like a sitemap.xml.
Cool tools like https://realfavicongenerator.net/ help you create the right set of favicon images and helps with the manifest.json
content. We just need a sufficiently large source PNG to upload.
Acceptable goal should be to get a perfect score on the PWA portion of Lighthouse.
In #85 (comment) Daniel and myself got into an off-tangent discussion about git submodules and how the stumptown-renderer
works. It's not urgent to solve now but it should definitely be on our active radar.
The goal should be, I think:
.md
file into master
Also, if you change the mdn-browser-compat-data
changes, that's a merge to master too and it should immediately update every page that has a BCD table in production.
stumptown-renderer
already has the ability to override which version of stumptown-content
it should build with. So, perhaps instead of stumptown-render --(specific sha of stumptown-content)--> stumptown-content
we make it the other way around. I.e. that stumptown-content
has a git submodule to stumptown-renderer
.
Then, in .travis.yml
, if all the tests pass, it could as a final step do something like this:
export STUMPTOWN_CONTENT_ROOT=`pwd`
cd stumptown-renderer
make deployment-build
if [ $branch = "master" ]; then
push-to-prod-s3-and-cloudfront client/build
fi
Note! I'm not convinced we update production just because there's a push to master
. It might be better to do something like (pseudo code):
if is_tag():
deploy_to_prod()
elif branch == "master":
deploy_to_stage()
elif trusted_user($PR_OWNER):
deploy_to_pr_submdomain()
else:
echo "Not deploing anything"
Since we rely on create-react-app
for the client
package, it comes with a robust and large-community maintained eslint
configuration.
For the record, it's tested automatically when you run cd client && yarn run build
but it's also in effect when you run cd client && yarn run start
(aka. make run-dev
).
But we have lots of other .js
files outside of client
that could benefit from eslint
validation.
On https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video for example. The HTML sent from the server to the browser looks like this:
<h2 id="Attributes">Attributes</h2>
Then, a piece of (jQuery??) code kicks in after the .js has been downloaded, parsed and executed (after document.onload
) that turns that into this:
<h2 id="Attributes">Attributes
<a href="/en-US/docs/Web/HTML/Element/video$edit#Attributes" class="button section-edit only-icon" rel="nofollow, noindex">
<svg class="icon icon-pencil" version="1.1" width="24" height="28" viewBox="0 0 24 28" aria-hidden="true" focusable="false"><path d="..."></path></svg>
<span>Edit</span>
</a>
<a class="local-anchor" href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video#Attributes" data-heading-id="Attributes">
<svg class="icon icon-link" version="1.1" width="24" height="28" viewBox="0 0 512 512" aria-hidden="true" focusable="false"><path d="..."></path></svg>
<span>Section</span></a>
</h2>
Let's not do that in stumptown.
Not only does it cause the page to "flicker" as the little section chain-icons suddenly appears. That makes the site feel "wobbly". It can also cause a re-paint (aka. recalculate style).
Note: Using Chrome Performance dev tools, with 6x CPU slowdown, I found 4!! blocks of "Recalculate style" in the first 3.5 seconds! It's very hard to exactly see how large the repaint is due to the added extra <a>
tags.
Either way, it should be avoidable. I.e. let's not modify the DOM shipped from the HTML from the server. At least not just to add section links.
I think (and hope) the permalink section can be done entirely with CSS.
If I remember correctly, the only reason we're using Babel to transform cli/index.js
into cli/dist/cli.js
, with Babel (and Webpack), is for the sake of module importing.
If we just switch from import minimist from "minimist";
to const minimist = require("minimist");
we might be able to run everything without transpilation in Node 10.
There might be other features that Babel allows us, but stuff like fat arrow functions etc are perfectly fine in Node >=10 for example.
Every pull request on Stumptown should create a full deployment of the whole site and this deployment should tie into the pull request.
Unfortunately, there is good solution for rendering in HTML strings without a wrapper element.
If you do:
let stuff = "<p>paragraph</p>";
<p dangerouslySetInnerHTML={{__html: stuff}}/>
you get...
<p><p>paragraph</p></p>
html-react-parser
is OKish but it adds 21KB to the bundle (8KB gzipped) which just too much at this early stage. I'd rather learn to live with a bunch of <div>
tags everywhere.
Basically, this issue is about keeping an eye on the above mentioned issue.
MDN uses the locales prefix technique. Those URLs are not going away. It's not a necessary truth but if Stumptown is ever to take over Kuma it would be immensely useful to have all URLs intact and not force everyone and everything to redirect to new URLs.
Having the locale in the URL is also very useful when doing static file deployment. Even if you wanted to, it wouldn't be feasible to have a piece of server code that decides which content to display based on cookies and user agent headers.
Stumptown-content has currently not yet nailed how it's going to do l10n but it's most likely going to be a tree where the locale is that root. Similar to MDN URLs.
I.e. not content/html/reference/elements/video/fr/prose.md
but content/fr/html/reference/elements/video/prose.md
See also: mdn/stumptown-content#75
Today, the ZillaSlab
font is loaded by this: https://github.com/mdn/stumptown-renderer/blob/75e2e48fa929e8c015a5eb9a7a8fb845b2710b59/client/src/index.js#L4
What happens is that it loads:
/static/media/zilla-slab-latin-700.aefbe7f1.woff2
/static/media/zilla-slab-latin-600.861f6710.woff2
/static/media/zilla-slab-latin-400.a4701e4e.woff2
Why do we need all three? Can it be reduced to just 1 .woff2
URL?
Also, is latin
really right for us? Stumptown-renderer doesn't have any Chinese content yet but it will have.
That third-party package uses font-display:swap;
which is good. But if we could somehow figure out which URL(s) is going to be needed I think it would be nice to add this as a preload
tag so that downloading of the font happens sooner.
woff2
is great but browsers that don't support WOFF2 could probably make do without web fonts.
If we need to we can simply extract what typeface-zilla-slab
package does and do it ourselves.
User story:
make install
, make run-server
etc.http://localhost:3000/docs/Web/HTML/Element/video/
stumptown/content/html/reference/elements/video/docs.md
in an editor like vim or vscode.This daemon that monitors file changes would need to be something that developers ideally run in the foreground. That way they can see if a typo, for example, breaks the rendering update. So it would be nice to make it all pretty and colorful.
We could also do what create-react-app's webpack dev server does which is to catch errors in the daemon, send them to the browser and instead of reloading the page, show the whole error fullscreen.
The initial prototype, of this project, attempted to do a similar locale redirect solution as is done on MDN. E.g. / --> 302 --> /en-US
The problem is that we can't redirect to any URL that doesn't match a real file. I.e. <build>/en-US/index.html
When we enable a service worker we should add some registration callback so that we can display a "toast" encouraging the user to refresh here and now instead of waiting for the next time they restart the browser tab.
mdn/stumptown-content#74 has been merged and mdn/stumptown-content#89 is forthcoming. At some point, we'll need to update the renderer to render those pages in compliance with the new recipe.
This is a follow up to mdn/stumptown-content#55.
We're currently using font-display:swap
(It's by default when using typeface-zilla-slab
)
It means it renders the default OS font (or whatever we configure as number 2 in font-family) until the webfont loads. If the webfont comes in after 1s the headers will flash as they change style.
An attractive alternative might be to try fallback
or optional
as it'll display nothing for the first 100ms which hopefully avoids the flash of unstyled text.
This site explains the differences: https://font-display.glitch.me/
Note, we're using web fonts of headlines only. Which might be an important factor as evaluate this.
The BCD table is always far down. So it shouldn't block the rendering path.
Also, we should totally consider to use an intersection observer so that we only bother to load it if the user scrolls it into view.
Goal: Smaller HTML, smaller index.json
, less CSS, less JS CPU work; still perfectly fine looking BCD table for those who scroll down.
__compat
stuff in the index.json
files should be omitted.preload
things in case the user eventually scrolls the BCD table into view.The renderer doesn't yet know how to render BCD tables from the data. This is one of the relatively complicated bits of rendering.
The result should look pretty much like the existing BCD tables on MDN (https://developer.mozilla.org/en-US/docs/Web/CSS/margin#Browser_compatibility) which is built using the Compat.ejs KumaScript macro. So in a sense this is a port of Compat as a React component.
There are a few bits which rely on JS in the page (opening the little disclosures for compat details).
Right now, it goes wrong because, I suspect, something's wrong with the react-router or the way Netlify butchers the URL in static html.
Either way, it needs to work eventually. In Kuma we had to do this: https://github.com/mozilla/kuma/blob/ea382cd58cb51ef39f0378bfce175df0e5e712bc/kuma/javascript/src/index.jsx#L65-L67
But we need to try to make sure interactive (iframes) examples also work. E.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1564963
At the moment the renderer doesn't render any additional prose sections.
I think, whenever you focus on hover over a code example <pre>
tag a little copy-to-clipboard icon should appear. Clicking it should copy the raw source code into the clipboard.
Generally in single page apps, where you use something like react-router
you just have 1 /index.html
(and some static assets). There you want URLs like /foo/bar/page
to actually serve up /index.html
and 200 OK
.
However, we have a file for every single possible valid URL. I.e. we have a file called /foo/bar/page/index.html
. It's a copy of /index.html
but with the content already server-side rendered into the HTML.
However, we need a solution so we can cope with typos. We don't want the custom stock 404 template you get with Netlify. We want our React code to kick in and run its own "Page Not Found" message.
Outside the browser, with curl for example, it's important that invalid URLs yield a 404 Page Not Found
HTTP error code. E.g. curl -v https://stumptown/foo/bar/pajge
should be 404
but look great.
What did you do?
1.Goto a RTL version of MDN ste. eg: https://developer.mozilla.org/fa/
2.Look at small arrows on side of Web Technologies; Learn web development; Developer Tools
3.Look at cat that shows "Help us build better CSS debugging tools!
Which CSS bugs are the worst?
Take the Survey"
4-Look at location of Arrow of "Sign Up now" button
5-From Refrences and Guides menu, look at more docs
What happened?
2-Small arrows are on their related texts.
3-Cat is located on it's related text.
4-Arrow of sign up now button is located on left side of button like LTR texts.
5-Three dots are beforer More docs menu in English text! Like:
...More docs
As result: localizers repeteated using ... before text on ntranslation of More docs as well as english
What should have happened?
2-Arrows should be mirrored and located on left side of their related texts on RTL languages. Look at samples:
LTR: Text --->
Correct RTL: <--- متن
Invalid RTL(Current status): م--تن->
3-Cat should be located on left side of it's related rectangle.
4-Small arrow on sign up now button should go to left side of button and txt should goto right side of button and arrow should be like <--- not --->
5-...More docs in English shold be More docs... and Localizers should move three dots aftre more docs like fixed english phrase.
Is there anything else we should know?
Currently, MDN main page is RTL but it's details like buttons, content of DIV tags are not RTL.
Bugzilla - https://bugzilla.mozilla.org/show_bug.cgi?id=1539793
bidi
mixinFor example, how the cli/render.js
mutates the rendered HTML string
This is fragile already and we intend to do more stuff in SSR such as replacing <head><meta>
tags and stuff.
stumptown-renderer
ships with a specific version of stumptown-content
as a git submodule. And this "relationship" is hardcoded in the Makefile
to the stumptown-content
as git submodule.
You're free to not use Makefile
and instead open it and copy-n-paste the various things the the Makefile
does. However, that can become tedious and complicated. It would be much better to be able to do something like:
STUMPTOWN_CONTENT_ROOT=/my/other/place/of/stumptown-content make build-content
Because they want to present code and output side by side, interactive examples like to have as much width as they can, and that's accommodated on MDN: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video
With a browser width of 1440px the example gets ~1000px, extending beyond the text which is limited to 42rem.
In mdn2 the interactive example seems to be restricted to the same width as the text (800px).
We're currently doing this:
<script id="documentadata" type="application/json">
{"title": "Video ...", ...
</script>
and
let documentData = null;
const documentDataElement = document.getElementById("documentdata");
if (documentDataElement) {
documentData = JSON.parse(documentDataElement.text);
}
First of all, should we use domelement.textContent
instead of domelement.text
??
Another approach, which might be faster is to use:
<script id="documentadata" type="application/json">
window.DOCUMENT_DATA = JSON.parse('{"title": "Video ...", ... ');
</script>
In this benchmark Henrik Joretag argues that it's faster to use a script tag and call JSON.parse()
in it. The blog post also mentions interesting topics around escaping the JSON string correctly.
This approach obviously is going to need to be reflected in the CSP headers. If we set our CSP headers in a Netlify config file, so it becomes real HTTP headers, then can we update the nonce in runtime with a <meta>
tag?
That last point about the CSP complexity might escalate the complexity to the point where this (possible) small performance boost isn't worth it.
We currently have a super basic CSS skeleton based on water.css plus our own dumping groud mdn.scss
where we've ported some necessary CSS stuff to display the pages correctly.
What would be nice is a ground-up solid CSS framework with a decent grid and a decent header and footer.
It should be responsive at least.
(This is not about the content)
We need to figure out the right/best way to use gettext
in the client code. Thankfully the client has very little English but it needs to to be localizable.
Setting up the tooling to something like mdn-l10n
might be best to do as a separate issue to reduce the size of the scope.
mdn
GitHub orgclient
appdocker-compose
to start all the thingssitemaps.xml
etc.)stumptown-experiment
git submodulehumans.txt
fileA declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.