fred-wang/texzilla

LALR Javascript LaTeX-to-MathML converter compatible with Unicode

TeXZilla

License

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

Description

TeXZilla is a Javascript LaTeX-to-MathML converter compatible with Unicode. It has performed as the fastest state of the art LaTeX-To-MathML converter according to recent research in this field (see [1]). This is still a work in progress and things may change in the future. Please report any bug you find to the issue tracker.

For a quick overview, you can try a live demo, install a Firefox add-on or try a Firefox OS webapp.

You can download a release archive or install an npm package.

Please read the wiki to get more information on how to integrate TeXZilla in your Web page or project as well as a description of the TeXZilla syntax. See also the examples/ directory.

Build Instructions

The following dependencies are required:

• coreutils, sed, curl, make, procps, grep
• xsltproc
• Python
• Jison.
• To run unit tests: slimerJS or phantomJS, bash. nodejs can be used to run the DOM-less tests.
• To generate the minified version TeXZilla-min.js: Google Closure Compiler.

On Debian-based Linux distributions, try sudo apt-get install coreutils sed curl make xsltproc python npm phantomjs bash closure-compiler and install Jison with npm install jison -g.

To build TeXZilla, run the tests and generate the minified version:

  ./configure
  make all
  make minify

Type make help for more commands.

References

[1] "Improving the Representation and Conversion of Mathematical Formulae by Considering their Textual Context" by M. Schubotz, et al. In: Proceedings of the ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL). Fort Worth, USA, June 2018. DOI:10.1145/3197026.3197058

texzilla's Issues

\hline is not supported

Is it a bug? Is there any alternative way to draw a horizontal line to separate rows in a matrix?

Add configuration to decide which kind of letters have automatic italic

Follow-up of #71

It seems this should be configurable:
http://milde.users.sourceforge.net/LUCR/Math/math-font-selection.xhtml#math-styles

The minified version declares undesired global variables

After application of Google Closure Compiler, new variables h, n, p and u are defined in TeXZilla-min.js. The content of the file should be wrapped in a function() to prevent these variables from being global.
Also, the "use strict" mode should only apply to that function scope to avoid concatenation problem.

Use a standard commonJS syntax for exit

Currently, we use three versions of "exit" for phantomjs, slimerjs and nodejs. They are used in unit-tests.js and commonJS.js

Ideally, the exit call should work with any CommonJS program.
http://wiki.commonjs.org/wiki/CommonJS

Height of region surrounding the equation

How does one change the height of the region surrounding a rendered MathML code?

Parse error for some matrix environment

The amsmath package define six environments for matrix:

• matrix,
• pmatrix,
• bmatrix,
• Bmatrix,
• vmatrix,
• Vmatrix.

For bmatrix, Bmatrix and Vmatrix we got a parse error.

Steps to reproduce

• Build the parser
• Open index.html with Firefox.
• Type in the box (you can replace bmatrix with Bmatrix or Vmatrix):

\begin{bmatrix}
a_11 & a_12 \\
a_21 & a_22
\end{bmatrix}

• Press 'TAB'.

Some TeX commands should be <mo movablelimits="true">

See the discussion on the MathML mailing list:
http://lists.w3.org/Archives/Public/www-math/2014Jan/0003.html

Errors in LaTeX commands for characters

\partial, \nabla, \mu, \nu maps to the incorrect characters.

This is because the W3C's unicode.xml has duplicate/wrong LaTeX mappings:
http://lists.w3.org/Archives/Public/www-math/2013Dec/0001.html

Add unit tests for all mathvariant mappings

Currently we have tests for latin and digits. We need some for greek, arabic and various exceptions.

Do not use multiple values for tabular attributes (columnalign etc)

From https://mathml-refresh.github.io/mathml/chapter3.html#presm.mtable :

In the above specifications for attributes affecting rows (respectively, columns, or the gaps between rows or columns), the notation (...)+ means that multiple values can be given for the attribute as a space separated list (see Section 2.1.5 MathML Attribute Values). In this context, a single value specifies the value to be used for all rows (resp., columns or gaps). A list of values are taken to apply to corresponding rows (resp., columns or gaps) in order, that is starting from the top row for rows or first column (left or right, depending on directionality) for columns. If there are more rows (resp., columns or gaps) than supplied values, the last value is repeated as needed. If there are too many values supplied, the excess are ignored.

The parser could convert them to single values on mtd elements.

Use explicit units

For example linethickness="0" should be linethickness="0px"

Improve Travis Continuous Integration

I've connected the TeXZilla repository to travis. After several attempts, I ended up doing "build only".

https://travis-ci.org/fred-wang/TeXZilla/builds

Ideally we should also:

1. run the tests with nodejs
2. run the tests with phantomjs
3. run the tests with slimerjs (not available as a package for Ubuntu precise)
4. run "make minify" (closure-compiler not available as a package for Ubuntu precise, but is in more recent version)

Prime sign treated as superscript

Consider the input "A'". TeXZilla.toMathMLString("A'") gives:

<math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>A</mi><mo>′</mo></msup><annotation encoding="TeX">A'</annotation></semantics></math>

Unfortunately, the macro \prime seems unsupported. Moreover, typing the Unicode character "’" gives a parse error. They are all workarounds I am aware of.

<math xmlns=\"http://www.w3.org/1998/Math/MathML\"><semantics><merror><mtext>Parse error on line 1:
\\(A’\\)
----^
</mtext></merror><annotation encoding=\"TeX\">A’</annotation></semantics></math>

Styling the page

Would it be fine if I styled this page to be a lot more visual friendly?

Convert dash (U+2D) to minus sign (U+2212)

Browsers do that internally but it's probably better to do it during conversion.

Abtract away the element/attribute structure

In general, the structure handled by TeXZilla is a string representing an element.

The serialization could be defered and we could instead handle a structure containing:

• an element name
• a list of (attribute, value) pairs on that element.
• a string representing the children

This will allow to:

• avoid duplicate attributes (not well-forme XML). As in itex2MML, this may happen with table attributes.
• avoid to use useless mstyle or mrow. For example the mathvariant or color attributes can just be set on token elements.

Do not introduce extra mstyle elements to set presentation attributes on token elements

Follow-up of #10

For example the mathvariant or color attributes can just be set on token elements.

test: \mathfrak{a}

Make parseLength support MathML namedspaces

parseLength currently only accepts some numeric length as in itex2MML. It might be helpful to support the MathML namespaces too. See http://www.w3.org/TR/MathML3/appendixa.html#parsing_length

Support itex2MML commands

That should almost be the case.

• Issues #12 and #5
• Need an option to treat xy as <mi>xy</mi>
• Need to write unit tests for each itex2MML command to ensure they are supported.

switch observer model in custom element?

I was thinking the other day that the observer model is kind of flawed for custom elements because the lifecycle is so wonky... for example, currently

document.body.innerHTML = `<la-tex>f(x)=\\sum_{n=-\\infty}^\\infty c_n e^{2\\pi i(n/T) x} = \\sum_{n=-\\infty}^\\infty \\hat{f}(\\xi_n) e^{2\\pi i\\xi_n x}\\Delta\\xi</la-text>`

Will fail to fire (as would I if you included this at the end of your document, probably) as the nodes already exist - so they aren't being treated 'as parsed'. I think I added this this way and just never added the workarounds.... But.... I think the simplest one is to actually just not use a mutationobserver directly at all and make use of the slotchange event which seems to happen regardless, consistently... Maybe we should switch to use that? Something kinda like this https://glitch.com/edit/#!/mathml-examples?path=js/elements/la-tex.js:21:5 (though, obvs we'd rework a little to include attrs correctly too here?) The above snip will work if you try it in the console on https://mathml-examples.glitch.me/foundation-expansion.html

If we want I can work this up and send a pull...

Add an option to throw an exception when parsing fails

This was reported by @r-gaia-cs. My proposal is to add an optional parameter aThrowExceptionOnError:

toMathMLString = function(aTeX, aDisplay, aRTL, aThrowExceptionOnError)
toMathML = function(aTeX, aDisplay, aRTL, aThrowExceptionOnError)

so that when aThrowExceptionOnError = true, TeXZilla will throw an exception instead of returning an merror element.

Tilde for a non-breaking space is not supported

The tilde character ~ should be treated as a non-breaking space. See the LaTeX FAQ. Original issue: josdejong/mathjs#1299

Input

x~y~z

Expected output

<mi>x</mi>
<mspace linebreak="nobreak" width="mediummathspace" />
<mi>y</mi>
<mspace linebreak="nobreak" width="mediummathspace" />
<mi>z</mi>

Actual output

<mi>x</mi>
<mo stretchy="false">~</mo>
<mi>y</mi>
<mo stretchy="false">~</mo>
<mi>z</mi>

Improve Test Coverage

We should have at least one test for each TeX command and even better each rule in the grammar.
https://github.com/fred-wang/TeXZilla/blob/master/unit-tests.js

We should check that most itex2MML commands are supported.

We should also have one test for each function of the public API / command line.

Strange value for lspace

Line 493 of TeXZilla.jison has

$$ = newTag("mpadded", $2, "width=\"0em\" lspace=\"-100%width\"");

The value for lspace is correct?

Strict Mode

$ git log HEAD -1
commit 71f92bc764761fb69db8030e5d548ddcc83d98ca
Author: Frédéric Wang <[email protected]>
Date:   Sun May 31 12:59:01 2015 +0200

    Travis: Just run "make build" without testing for now...
$ texzilla
/usr/local/lib/texzilla-js/TeXZilla.js:876
        function lex() {
        ^^^^^^^^
SyntaxError: In strict mode code, functions can only be declared at top level or immediately within another function.
    at exports.runInThisContext (vm.js:73:16)
    at Module._compile (module.js:443:25)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)
    at startup (node.js:124:16)
    at node.js:842:3
$ node --version
v0.11.14

Prime and superscript on the same character cause a parse error

For example q'^2

Can't resolve 'webserver'

Hey,
when packing our site with webpack which depends on texzilla, we run into #53 . If we then add "system" as a dependency, we get this warning message instead:

Module not found: Error: Can't resolve 'webserver' in './node_modules/texzilla'
 @ ./node_modules/texzilla/TeXZilla.js
...

It seems like both this issue and #53 still need to be fixed. In #53 it sounds like you are not planning to fix it here - so maybe somewhere else?

Java JDK7 Rhino ScriptEngine issues

Hello,
I've been using TeXZilla as a library in Java ScriptEngine API,
(http://docs.oracle.com/javase/7/docs/technotes/guides/scripting/programmer_guide/)
I found a couple of issues with TeXZilla 0.9.7 when running on the following engine:

JS engine name: Rhino
version: Rhino 1.7 release 0.7.r2.2.el6 2010 08 21
language version: 1.7

This is the default engine for Java 1.7

1. Math object is sealed, cannot add the log2 property
2. Object.getPrototypeOf doesn't exist

I've attached a quick patch that fixes the issues.

server-side LaTeX-to-MathML Drupal plugin

https://drupal.org/ has a module to use nodejs:

https://drupal.org/project/nodejs

This means that TeXZilla could be executed server-side to perform LaTeX-to-MathML conversion.

Automate release of npm package

I've just published an npm package:

https://www.npmjs.org/package/texzilla

The release.sh script should be able to automate updates of npm package.

Improve error message

The default error message don't help very much.

Step to reproduce

1. Try to parse \frac{1}.

It show

Parse error on line 1:
\frac{1}
--------^
Expecting '{', 'LEFT', 'OPFS', '.', 'BIG', 'BBIG', 'BIGG', 'BBIGG', 'BIGL', 'BBIGL', 'BIGGL', 'BBIGGL', 'NUM', 'TEXT', 'A', 'F', 'MI', 'MN', 'MO', 'OP', 'OPS', 'OPAS', 'MS', 'MTEXT', 'HIGH_SURROGATE', 'BMP_CHARACTER', 'OPERATORNAME', 'MATHOP', 'MATHBIN', 'MATHREL', 'FRAC', 'ROOT', 'SQRT', 'UNDERSET', 'OVERSET', 'UNDEROVERSET', 'XARROW', 'MATHRLAP', 'MATHLLAP', 'MATHCLAP', 'PHANTOM', 'TFRAC', 'BINOM', 'TBINOM', 'PMOD', 'UNDERBRACE', 'UNDERLINE', 'OVERBRACE', 'ACCENT', 'ACCENTNS', 'BOXED', 'SLASH', 'QUAD', 'QQUAD', 'NEGSPACE', 'NEGMEDSPACE', 'NEGTHICKSPACE', 'THINSPACE', 'MEDSPACE', 'THICKSPACE', 'SPACE', 'MATHRAISEBOX', 'MATHBB', 'MATHBF', 'MATHBIT', 'MATHSCR', 'MATHBSCR', 'MATHSF', 'MATHFRAK', 'MATHIT', 'MATHTT', 'MATHRM', 'HREF', 'STATUSLINE', 'TOOLTIP', 'TOGGLE', 'BTOGGLE', 'TENSOR', 'MULTI', 'BMATRIX', 'BGATHERED', 'BPMATRIX', 'BBMATRIX', 'BVMATRIX', 'BBBVMATRIX', 'BVVVMATRIX', 'BSMALLMATRIX', 'BCASES', 'BALIGNED', 'BARRAY', 'SUBSTACK', 'ARRAY', got 'EOF'\frac{1}

Comparison with LaTeX

$ cat e.tex
\documentclass{article}
\begin{document}
$\frac{1}$
\end{document}
$ pdflatex -interaction=nonstopmode e
...
! Missing } inserted.
<inserted text> 
                }
l.3 $\frac{1}$

! Too many }'s.
\frac #1#2->{\begingroup #1\endgroup \over #2}

l.3 $\frac{1}$
...

Solution

Replace JISON parser.parserError as suggest by @zaach here and here.

Use non-breaking space in token elements

Currently, TeXZilla collapses whitespace in token elements:
https://github.com/fred-wang/TeXZilla/blob/master/TeXZilla.jison#L235

Perhaps instead of calling trim(), the leading/trailing space should be replaced by a  . This can be done by modifying the trim polyfill here:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/Trim#Compatibility

So

$1.replace(/^\s+|\s+$/g, "\u00A0").replace(/\s+/g, " ");

will work I think.

Do not use the mathvariant attribute

It can be replaced with direct use of the transformed character. #10 (and maybe #61) might help here.

Escape Text

In TeXZilla.jison we have

function escapeText(aString)
{
  /* Escape reserved XML characters for use as text nodes. */
  return aString.replace(/&/g, "&amp;").replace(/</g, "&lt;");
}

Should we replace > with >?

Use `phantom.injectJs` instead of `require`

The file commonJS.js exists because of the line

var parser = require("./TeXZilla").parser;

in unit-tests.js.

What about replace this line with

phantom.injectJs("./TeXZilla.js");

and remove the file commonJS.js?

Remove unicode.patch when unicode.xml is fixed upstream

After #5 unicode.xml is patched:
http://lists.w3.org/Archives/Public/www-math/2013Dec/0001.html

The patch is a quick workaround and I'm not sure it's very good. Hopefully @davidcarlisle will update unicode.xml soon.

Do not used named spaces for MathML lengths

cc @distler

TeXZilla uses some named spaces for MathML lengths ( w3c/mathml#55 (comment) )

Although it is perfectly valid MathML, some people from the MathML CG proposed to remove them from MathML Core ( w3c/mathml#63 w3c/mathml#75 ) so that we align more with CSS.

Not sure this is going to happen but I think it is safe to use equivalent em values instead.

Arabic characters are not accepted

Using Arabic characters (either from regular or math blocks), cause a parse error, eg:

ط^٢

No problem with digits, or Latin math characters.

Improve lexical analysis

Do not make uppercase greek letter italic

For example \alpha generates α which has automatic mathvariant=italic. LaTeX does not seem to do that by default so we should add an explicit mathvariant="normal".

mtext test fail

This depends of #31

$ make tests
....
Test 26... FAIL
TeXZilla.toMathMLString, unexpected result:
  Actual: '<math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mtext> x y </mtext><annotation encoding="TeX">\\mtext{  x   y  }</annotation></semantics></math>'
  Expected: '<math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mtext> x y </mtext><annotation encoding="TeX">\\mtext{  x   y  }</annotation></semantics></math>'
...

[WebKit/Blink] Can retrieve (La)TeX

TeXZilla can't retrieve (La)TeX because in WebKit/Blink DOM Element.children does not work on MathML elements.

TeXZilla streamfilter does not work with slimerjs

See laurentj/slimerjs#188

Add LaTeX commands for Arabic mathvariants

Currently, we have \\mathbb, \\mathbf \mathtt etc We don't have things for the Arabic mathvariants at least.

Merge master and gh-pages

Was wondering since github now allows any branch to be hosted and the gh-pages is pretty much the same as the master branch. Perhaps it'd be a good idea to have one of these branches deleted (preferrably gh-pages).

Structure not converted to string

This is related to issue #10 and introduced with r-gaia-cs@635732f.

Some structures/objects aren't converted to string properly:

Test 19... FAIL
  Actual: '<math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mroot>[object Object][object Object]</mroot><annotation encoding="TeX">\\sqrt[3]x</annotation></semantics></math>'
  Expected: '<math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mroot><mi>x</mi><mn>3</mn></mroot><annotation encoding="TeX">\\sqrt[3]x</annotation></semantics></math>'

Can't resolve system

I installed TeXZilla using NPM. When I tried to import it, this error appeared:

./node_modules/texzilla/TeXZilla.js
Module not found: Can't resolve 'system' in '/home/m93a/Dokumenty/Vývoj/punk-tex/node_modules/texzilla'

Improve expression grouping using operator precedence

See http://www.w3.org/Math/MathML3/chapter3.html#id.3.3.1.3 for a discussion of grouping of sub-expressions.

Expression grouping was removed in 2f3e5f4 because it didn't work very well and was a serious performance burden for Jison.

I believe in general, correct operator grouping is not possible because one has to infer some semantics from the TeX source , especially when fences are involved like in y = { [0,1) ; (2,3] }^2.

However, I think it would still be possible to improve the grouping via operator precedence:

• do this in newMrow instead of relying on the grammar's operator precedence
• only do this for a restricted set of operators like unary/binary operators and relations.
• fences should not be involved or only at the beginning and end of the mrow

Cleanup the this.parseError code

TeXZilla generates the following code

    if (true) {
         this.parseError = parseError;
     } else {
    }

in order to avoid error with Object.getPrototypeOf in Rhino (issue #38). This could be cleanup a bit further.

Enhancements to the commonJS command line API

Currently, the commonJS API only exposes the getTeXSource, toMathMLString and toMathML functions. When commonJS is more standardized it would be useful to extend the API. Concrete use cases:

• allow to use TeXZilla as a stream filter.
• allow to run TeXZilla as a Web service.

http://wiki.commonjs.org/wiki/System/1.0
http://wiki.commonjs.org/wiki/JSGI