Test repository for EPUB3, maintained by the EPUB3 WG
Home Page: https://w3c.github.io/epub-tests/
License: Other
This repository contains tests to validate the implementability of the W3C's EPUB 3.3 specifications, specifically core
EPUB 3.3 (the spec for the EPUB format itself) and
EPUB Reading Systems 3.3 (the spec for applications that read EPUB files). Our
objective is to test every normative statement (that is, every
MUST or SHOULD or MAY, etc.).
The test reports describe all the tests, including implementation results and how to contribute.
Will prepare a test for the Text-to-Speech section.
I will test
Reading systems MUST NOT use ancillary manifest information contained in the ZIP archive or in the manifest.xml file [epub-33] for processing an EPUB publication.
from https://w3c.github.io/epub-specs/epub33/rs/#sec-container-metainf-manifest.xml
I will test
MUST treat any OCF ZIP container that uses compression techniques other than Deflate [rfc1951] as in error.
from https://w3c.github.io/epub-specs/epub33/rs/#confreq-zip-comp
Planning to test:
In the absence of this information in a Publication Resource, Reading Systems MUST NOT assume either the language or the base direction of that resource from information expressed in the Package Document
This whole section needs tests:
https://w3c.github.io/epub-specs/epub33/rs/#sec-epub-rs-i18n
There are some similar tests in https://w3c.github.io/epub-tests/#sec-internationalization-data, but more are needed to satisfy this section.
Screenshot from Apple iBooks / MacOS Books.app (same broken rendition with Thorium):
The rationale for including the Acid test in the EPUB test suite was mentioned in this PR #7
I also made a version of the Acid2 test, as that seemed one way to test the requirement about reading systems with a viewport: "If it has a Viewport, it MUST support visual rendering of XHTML Content Documents as defined in § 4.3.1 CSS Conformance."
This Acid test was indeed a useful exercise that offered a vivid visual demonstration of the shortcomings of some competing CSS implementations in web browsers and other rendering engines. However I think the value of Acid test in the context of the EPUB test suite is arguably debatable because it is not clear what feature is tested exactly. Furthermore, reading systems typically apply a default "user agent" stylesheet (+ additional overrides based on user preferences) that cause different rendering than vanilla web browsers. For example, pagination using CSS columns usually involves overriding margins, etc.
My recommendation would be to eliminate the Acid test from this test suite.
I will add test for following statement.
Reading systems MUST NOT fail when encountering configuration files in the META-INF directory not listed in Reserved files [epub-33].
https://w3c.github.io/epub-specs/epub33/rs/#sec-container-metainf-inc
There is a statement in the RS document:
If any of the properties attribute's values do not include a prefix [epub-33], reading systems MUST use the prefix URL "http://idpf.org/epub/vocab/package/item/#" to expand the values.
@dlazin wonders whether this (and similar) statements can be tested in the first place.
I tried to come up with a testing strategy and, before I put it into the test suite, I wanted to check whether this is indeed a good test.
Version 1. I create a simple test with a single content file, but with a package document of the form:
<package xmlns="http://www.idpf.org/2007/opf" xmlns:epub="http://www.idpf.org/2007/ops"
version="3.0" xml:lang="en" unique-identifier="pub-id"
prefix="myown: http://idpf.org/epub/vocab/package/item/#">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
...
</metadata>
<manifest>
...
<item id="nav" properties="myown:nav" href="nav.xhtml" media-type="application/xhtml+xml" />
</manifest>This is not a valid EPUB insofar as there is a statement in the core whereby one MUST NOT define a prefix for a default vocabulary. Epubcheck flags this, and that is fine.
Version 2. I modify the EPUB to use prefix="myown: http://example.org/#".
My expectation would be that a RS should open the first version without further ado, ie, the TOC should be identical to the 'classic' case when no prefix is used. On the other hand, the second case should either provide no TOC or some sort of a default, RS-dependent TOC (eg, a list of file names).
Is this correct? There is an uncertainty whereby the first version is, as I said, not 100% correct, so a RS might refuse it, too...
I'll write a test for this normative statement:
https://w3c.github.io/epub-specs/epub33/rs/#sec-linked-records
In the case of a linked metadata record [EPUB-33], Reading Systems MUST NOT skip processing the metadata expressed in the Package Document and only use the information expressed in the record.
The test also relates to:
https://w3c.github.io/epub-specs/epub33/core/#sec-linked-records
EPUB Creators MAY provide one or more linked metadata records to enhance the information available to Reading Systems, but Reading Systems may ignore these records.
Taking care of those
This test the normative statement found at:
https://www.w3.org/TR/epub-rs-33/#confreq-rs-data-urls
Reading Systems MUST prevent data URLs [RFC2397] from opening in top-level browsing contexts.
I will do a basic MO test that may serve as a starting point for additional tests. Avoiding a mega PR I will do one with a single test to agree on the 'styles' (naming, file patterns, etc).
Cc: @marisademeglio I used your MO tests on the idpf test suite, to extract an individual, smaller test.
At the moment, all our tests are related to MUST statements but, eventually, we may also have tests for SHOULD-s. The distinction is significant for reporting: for the CR->PR transition, we will have to report the required number of implementations, whereas no such requirement exists for SHOULD-s. This difference must be clearly noted on the test report and this leads to the necessity to provide this information in the package metadata.
I did not find any obvious way to represent this information in the dcterms vocabulary. With a bit of an abuse of the existing vocabularies, I have come up with two different approaches.
One is to look at the tests as a collection in terms of the EPUB metadata vocabulary and add the following:
<meta property="belongs-to-collection">must</meta>The value for the belongs-to-collection term is must or should.
If this is considered to be an abuse, we could define our own "vocabulary":
<package … prefix="testType: https://www.w3.org/ns/test_types/">
...
<meta property="testType:statement">must</meta>
...again with the value of the terms defined as must or should.
In both cases the missing metadata value should be taken as a must by default (this means we do not require reworking all the current tests...
Any better approach?
Cc: @dlazin @mattgarrish @wareid @dauwhe
As for the UI, my idea is to group the SHOULD tests at the end of the various tables, label all the rows with "M" or "S" (also, possibly display them with a slightly different background color). The point is that it should be clear to, mentally, separate the two category of tests when reviewing the results.
In addition to if an audio clip is played from clip start to clip end, it also tests if the text is highlighted and if the reading system has a button to start playback (which is technically not a requirement, if the RS just started playback automatically we would still say it supports MO).
The relevant commit is 631baf0
The title tells it all... I can easily extend the automatic generation scripts by collecting the creator metadata from the package document alongside the attributes that are to be extracted anyway. The script would then create an acknowledgement section into the test description document.
Should we do this?
I will take care of the missing i18n tests (ie, rows 2-5 of the missing tests' sheet).
https://w3c.github.io/epub-tests/#container-default-rendition_arbitrary tests package documents in an arbitrary directory (FOO/BAR/package.opf), but we don't have any tests of packages with arbitrary filenames (foo.opf and maybe foo.bar or just foo).
That should be added as an additional test case for https://w3c.github.io/epub-specs/epub33/rs/#container-default-rendition ("A Reading System MUST, by default, use the Package Document referenced from first rootfile element to render the EPUB Publication.")
Testing the statement:
MUST NOT show list item numbering on nav elements when presenting them outside of the spine, regardless of support for CSS
This requirement does not have any tests.
I will test the following normative statement:
https://www.w3.org/TR/epub-rs-33/#confreq-ocf-fobfus
Reading Systems MUST support deobfuscation of resources as defined in Resource Obfuscation [EPUB-33].
I take care of completing the missing tests in §9 Loading the media overlay
I will create tests for the Rendition flow properties (see also in the content document).
Note, for the record, that these tests are not really my own. I looked at the old IDPF test suite for 3.0 to see whether and how those tests can be repurposed by adapting them to 3.3 and the tests on rendition seem to be the simplest to convert manually (those were written by @mattgarrish). The other tests may be more difficult to repurpose, their granularity vs. the document features is different.
@mattgarrish @dauwhe, do you see any issues doing this?
Will look at testing of the Embedded audio and video section of the RS spec.
https://github.com/w3c/epub-tests/blob/main/CONTRIBUTING.md
Contributions to this repository are intended to become part of Recommendation-track documents governed by the W3C Patent Policy and Software and Document License. To make substantive contributions to specifications, you must either participate in the relevant W3C Working Group or make a non-member patent licensing commitment.
From what I understand, the tests themselves are not part of Recommendation-track documents, and the licenses used for tests elsewhere in W3C (see 1 and 2) don't seem to be using the Software and Document License.
I will test this:
https://w3c.github.io/epub-specs/epub33/rs/#sec-pkg-doc-relative-urls
epub-tests/tests/pub-foreign_bad-fallback/EPUB/package.opf
Lines 20 to 27 in 9a023ce
In this case, the resulting reading order (i.e. computed spine, once the fallback chain is resolved) is empty if the reading system follows the cascade of unsupported media types. In some reading systems this may cause a crash / broken user interface, in others this may result in an empty viewport ... what consitutes a pass vs. fail?
Although we concentrated on RS tests, we will have to report on three rec-track documents. Putting aside the issue on whether the current test harness can cover all requirements in the various document, the current report gives very little clue on the "origin" (apart from links to the relevant statement). We could:
dcterms:isReferencedBy statements). (Many test will have references to several documents.)Do we need something like that? If yes, do we need the concept of "primary" document?
Statements in §6.4 on not allowing CCScripts to modify the DOM
I added metadata to https://github.com/w3c/epub-tests/blob/main/tests/confreq-rs-xml-nval_invalid/OPS/package.opf (and renamed it, formerly xml-non-validating-parser-002), but it wasn't clear to me what's invalid in that test. Please clarify the test description and name, and possibly rename the test directory (and update test references in the spec).
By comparison, https://github.com/w3c/epub-tests/blob/main/tests/confreq-rs-xml-nval_unclosed/OPS/package.opf (same normative statement) is clear (but it could also be called "invalid," which is why "invalid" isn't good enough).
These three normative statements need tests:
https://w3c.github.io/epub-specs/epub33/rs/#confreq-rs-pkg-duplicate-item-rendering
I am not claiming these tests yet; this is just a tracking bug. Anyone may snag this bug so long as it remains unassigned.
In the current table view of all the issues the Must issues all look fine, but the "Should" issues seem like they are grayed out which makes it hard to read and fails WCAG 2.4.3 for normal text size fonts.
Since we have the column for "Req" containing Must/Should etc. we don't really have to also visually gray out these rows.
Now perhaps just the words "Must" "Should" etc could had a different background
Black on Red - Must
Black on Magenta - Should
Black on Yellow - May
as shown here:
The current tests cover how that the Reading system will find the listed in the spine. Necessary, but not enough; I intend to add some more tests that on the way the RS resolves URL-s within the HTML file, i.e., the way they use URL resolving for the webview they use.
Line 81 in fcd9a4f
vs.
epub-tests/tests/cnt-css-fonts/EPUB/package.opf
Lines 3 to 15 in 447aa8d
Suggestion: add <dc:publisher>W3C</dc:publisher> to match the metadata of the OPDS feed.
Reading System do not have to use or present linked resources, even if they recognize the relationship defined in the rel attribute.
Which reading systems use linked metadata records?
What record formats do they support? ONIX? MARC? Schema.org in some serialization?
In the case of a linked metadata record [EPUB-33], Reading Systems MUST NOT skip processing the metadata expressed in the Package Document and only use the information expressed in the record. Reading Systems MAY compile metadata from multiple linked records; they do not have to select only one record.
When it comes to resolving discrepancies and conflicts between metadata expressed in the Package Document and in linked metadata records, Reading Systems MUST use the document order of link elements in the Package Document to establish precedence (i.e., metadata in the first linked record encountered has the highest precedence and metadata in the Package Document the lowest, regardless of whether the link elements occur before, within or after the package metadata elements).
We could override the book title in a linked resource, and see if it works :)
Reading Systems MUST ignore any instructions contained in linked resources related to the layout and rendering of the EPUB Publication.
What are we trying to prevent here? Would the package doc link to, um, another package document that contains conflicting FXL metadata?
Statements in §6.4.
I will try to come up with those.
@dlazin, this is a new category of tests, I believe; we may want to agree on the naming upfront (and not at PR time). My proposal:
Coverage Label: Scripting, to be added between the current labels of Content Documents and Navigation Documents
Test names: scr-epubReadingSystem-* where '*' will probably say "properties" and "features" (I believe I can get away with two tests, but that is still open)
The epubReadingSystem is a mouthful, but that is the nature of the beast... I was wondering to use something like ers, but that would mean trading mouthful to cryptic. I would choose mouthful... WDYT?
There are four normative statements of this flavor:
If the property attribute's value does not include a prefix, Reading Systems MUST use the prefix URL "http://idpf.org/epub/vocab/package/meta/#" to expand the value.
https://w3c.github.io/epub-specs/epub33/rs/#meta
They need tests, but how do we test this? What does passing look like? What does failing look like?
The description for https://w3c.github.io/epub-tests/#confreq-rs-foreign_image is:
An HTML content file contains a PSD image, with a manifest fallback to a PNG image. This tests fallbacks for resources that are not in the spine.
The first half makes sense as a test for https://w3c.github.io/epub-specs/epub33/rs/#confreq-rs-foreign, but I don't understand how the spine is involved. Is the spine part irrelevant, or, if not, should this perhaps be two separate tests (image fallback and resource-not-in-spine fallbacks)?
See https://github.com/w3c/epub-tests/blob/main/tests/confreq-rs-foreign_image/OPS/package.opf for a refresher of what this test is actually doing :)
This issue came up during the 2nd vF2F day: we may want to add a metadata item into the package.opf file, namely:
<dc:rights>https://www.w3.org/Consortium/Legal/2015/copyright-software-and-document</dc:rights>The reason is that some reading systems will have to put a copy of the tests into another public space for testing, and the questions on rights may arise.
(Caveat: we would have to add this line to all opf files. Wonder whether this could be automatized; I am willing to look into this in case we otherwise agree.)
An EPUB Reading System "MUST be a conformant processor as defined in XML-NAMES."
Looking at XML Names, we find
To conform to this specification, a processor MUST report violations of namespace well-formedness, with the exception that it is not REQUIRED to check that namespace names are URI references [RFC3986].
It follows that in a namespace-well-formed document:
- All element and attribute names contain either zero or one colon;
- No entity names, processing instruction targets, or notation names contain any colons.
I suppose this means we write some tests with element names like p::p. But this all means that an EPUB Reading System is obligated to report a very technical error to the end user.
Is this our goal? My perhaps-naive view is that [1] we don't want reading systems to mess up the namespaces they already have to deal with and [2] we want to be able to use namespaces to change the rendering of the document via CSS.
Test this normative statement:
https://w3c.github.io/epub-specs/epub33/rs/#confreq-css-rs-fonts
To support CSS, a reading system:
MUST support [truetype], [opentype], [woff] and [woff2] font resources referenced from @font-face rules [css-fonts-4].
I will test
MUST treat any OCF ZIP container that specifies the [zip] file is split across multiple storage media as in error.
from https://w3c.github.io/epub-specs/epub33/rs/#confreq-zip-mult
Test this normative statement:
https://w3c.github.io/epub-specs/epub33/rs/#confreq-svg-rs-css
MUST, if it has a viewport, support the visual rendering of SVG [content documents] using CSS as defined in Styling [svg]
I will test following statement:
MUST support the ZIP64 extensions defined as "Version 1"
from https://w3c.github.io/epub-specs/epub33/rs/#confreq-zip-64
Brady mentioned the idea of a JSON file for test structure and metadata that would be processed into a test EPUB.
I wonder if XML might be more appropriate, just because almost all EPUB documents are XML-based. I was playing around with a test for a package version attribute less than three.
The idea here is you can specify exactly what you want to test, but some script will fill in all the boilerplate.
<package xmlns="http://www.idpf.org/2007/opf" version="0" xml:lang="en" unique-identifier="q">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
<dc:title>Package Version Zero 001</dc:title>
<meta name="assert" content="This test checks if a reading system will open an EPUB with package version attribute less than 3" />
<meta name="test" content="test passes if this text is visible" />
<meta name="help" href="https://w3c.github.io/epub-specs/epub33/rs/#confreq-rs-backward-epub"/>
&date;
&identifier;
&language;
</metadata>
&manifest;
&spine;
</package>Here, dc:title is the name of the test. The entities can be expanded to create the needed boilerplate.
meta name="assert" describes the test (this is the syntax used by CSSWG tests).
meta name="test" describes the criteria for the test passing--in this case, just that the EPUB opened. This text would be put in the HTML file for the EPUB.
meta name="help" is the link to the statement in the specification we are testing (this is the syntax used by CSSWG tests).
Similarly, we could use the same idea for HTML-based tests. Create the file we need to test with the relevant metadata, and then script into an EPUB. Example soon.
The use case is for such as mathml support some reading system has dependency on web browser to support so we will need a way to present "mathml can be supported on Safari" and "mathml can not be supported on chrome" for some web based reading system. I wonder if there is a way to present this difference.
I will test the following normative statement:
https://w3c.github.io/epub-specs/epub33/rs/#sec-epub-rs-external-links
Reading Systems SHOULD open links that resolve outside the EPUB Publication in a new browser instance to ensure that the security and privacy controls of the browser are available to users.
Apple Books does not display the navigation (Table of Contents) unless there are at least two links; I assume the thinking is that there's no need for navigation if you're already looking at the only thing. That's sensible UI logic, but does seem to contravene this normative statement:
https://w3c.github.io/epub-specs/epub33/rs/#confreq-nav-toc-access
To process the EPUB Navigation Document, a Reading System:
- MUST provide users access to the links and headings in the toc nav element [EPUB-33].
In a previous PR, I modified all the nav tests to have at least two links so that they would work in Apple Books, but we should have a separate single-link test for this statement.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.