nsoiffer / mathcat Goto Github PK
View Code? Open in Web Editor NEWMathCAT: Math Capable Assistive Technology for generating speech, braille, and navigation.
License: MIT License
MathCAT: Math Capable Assistive Technology for generating speech, braille, and navigation.
License: MIT License
Will MathCAT be adding support for the named entity definitions in the MathML DTD (⁡, ⅆ, ⅇ, ∬, ⁢, ±, etc.)?
I started creating test sets of MathML to look at the ClearSpeak and braille output compared to that produced by the SRE and noticed MathCat returns "XML parsing error at xx: {UnknownNamedReference}" when the provided MathML includes named entities.
For examples, please see:
Title pretty much say it all. It just says "zoom out; speech for math". Sometimes it repeats "zoom out".
For probability, an expression often repeats "P(...)". If the arg is simple, the canonicalization decides it is a function. But if it isn't (e.g., "P(A∩B)", then it comes up with "times".
Perhaps a more global search (in is_single_arg()
) to see if any of the "P(...)" inside of math
have a simple arg, and if so consider all of them to be function call. An optimization would be store a Vec
of found function names in CanonicalizeContext
so the more global search only happens once. The stored value would be "(name, bool)".
Another thing to do is to have a subject area specific list of probable function names. Maybe the current list has subject area "General" which is picked up by all (almost all?) subject areas.
I have encountered a case where the call to set_mathml leads to it altering the MathML to invalid MathML. Should get_braille then be called it will lead to an error. Below is the MathML I pass to the set_mathml call:
<math xmlns="http://www.w3.org/1998/Math/MathML" alttext="mml<mtable columnalign="left" equalrows="true" equalcolumns="true"/>(mml<mtr columnalign="left"/>mml<mtd columnalign="left"/>(bb I = bb text(Prt ) bb text(Write the simple interest formula) .) mml<mtr columnalign="left"/>mml<mtd columnalign="left"/>(text( ) = bb 500 (bb 0.03) (bb 3) text( ) bb text(Substitute ) bb 500 bb text( for ) bb P text( )_, bb 0.03 bb text( for ) bb r , bb text( and ) bb 3 bb text( for ) bb t .) mml<mtr columnalign="left"/>mml<mtd columnalign="left"/>(text( ) = 45 text( ) bb text(Multiply) .))"><mtable columnalign="left" equalcolumns="true" equalrows="true"><mtr columnalign="left"><mtd columnalign="left"><mstyle mathvariant="bold"><mo> </mo></mstyle><mi>I</mi><mo> </mo><mo>=</mo><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext>Prt </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext>Write the simple interest formula</mtext></mrow><mo> </mo><mo>.</mo></mtd></mtr><mo> </mo><mtr columnalign="left"><mtd columnalign="left"><mrow><mtext> </mtext></mrow><mo> </mo><mo>=</mo><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>500</mn><mo> </mo><mrow><mo>(</mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>0.03</mn><mo>)</mo></mrow><mo> </mo><mo></mo><mo> </mo><mrow><mo>(</mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>3</mn><mo>)</mo></mrow><mo> </mo><mrow><mtext> </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext>Substitute </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>500</mn><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext> for </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mi>P</mi><mo> </mo><msub><mrow><mtext> </mtext></mrow><mo>,</mo></msub><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>0.03</mn><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext> for </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mi>r</mi><mo> </mo><mo>,</mo><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext> and </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>3</mn><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext> for </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mi>t</mi><mo> </mo><mo>.</mo></mtd></mtr><mo> </mo><mtr columnalign="left"><mtd columnalign="left"><mrow><mtext> </mtext></mrow><mo> </mo><mo>=</mo><mo> </mo><mn>45</mn><mo> </mo><mrow><mtext> </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext>Multiply</mtext></mrow><mo> </mo><mo>.</mo></mtd></mtr></mtable></math>
In the altered MathML which is returned the msub only has a single child element when two are required.
I tried to create a minimal example <math><msub><mtext> </mtext><mo>,</mo></msub></math>
this though is correctly handled with the msub being converted to a prescript. So I am still to find the minimal example to cause this issue.
The braille is rendered correctly, but MathCAT apparently crashes while producing the speech output.
The example is taken from
vector-example.zip
https://en.wikipedia.org/wiki/Vector_space
In fixing #60, I made them be some (new) arrow code points.
These need to be handled for Nemeth and UEB translations.
There are some similar arrows, so they maybe get coded the same. Otherwise, they need to be dealt with as "arrow over arrow" constructs.
The Nemeth output for directly-over and directly-under expressions modified by a horizonal bar results in "dots-36 dots-36 dots-36 dots-36" instead of simply using the Nemeth cell for horizonal bar: dots-156 (Rule XIV - Modifiers near bottom of page 98). See examples in Rule XIV Section 86 a. The Five-Step Rule for Transcribing Modified Expressions pages 99-101; Partially shown here:
For the expression:
MathCat produced this Nemeth:
⠐⠷⠩⠤⠤⠤⠤⠀⠻⠹⠁⠌⠃⠼⠐⠾⠣⠀⠤⠤⠤⠤⠀⠻
The expected Nemeth was:
⠐⠷⠩⠱⠻⠹⠁⠌⠃⠼⠐⠾⠣⠱⠻
(also note no spaces between the directly-over indicator ⠣
or directly-under indicator ⠩
, the horizontal bar modifier ⠱
, and the termination indicator ⠻
)
\underline{(}\frac{a}{b}\overline{)}
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
<mstyle displaystyle="true" scriptlevel="0">
<munder>
<mo stretchy="false">(</mo>
<mo accent="true">―</mo>
</munder>
<mfrac>
<mi>a</mi>
<mi>b</mi>
</mfrac>
<mover>
<mo stretchy="false">)</mo>
<mo accent="true">―</mo>
</mover>
</mstyle>
</math>
The following MathML generated an 'index out of bounds' panic in interface.rs:
<math xmlns="http://www.w3.org/1998/Math/MathML">
<mrow>
<mi>
<mglyph fontfamily="my-braid-font" index="2" alt="23braid" />
</mi>
<mo>+</mo>
<mi>
<mglyph fontfamily="my-braid-font" index="5" alt="132braid" />
</mi>
<mo>=</mo>
<mi>
<mglyph fontfamily="my-braid-font" index="3" alt="13braid" />
</mi>
</mrow>
</math>
thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', src\interface.rs:488:27
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: Command failed: mathcat.exe <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi><mglyph fontfamily="my-braid-font" index="2" alt="23braid"/></mi><mo>+</mo><mi><mglyph fontfamily="my-braid-font" index="5" alt="132braid"/></mi><mo>=</mo><mi><mglyph fontfamily="my-braid-font" index="3" alt="13braid"/></mi></mrow></math>
thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', src\interface.rs:488:27
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
at checkExecSyncError (node:child_process:828:11)
at execFileSync (node:child_process:863:15)
at callMathCAT (C:\Users\richw\dev\brailleMathTests\main2-tests0001b-0.1.23.js:78:24)
at getMathResults (C:\Users\richw\dev\brailleMathTests\main2-tests0001b-0.1.23.js:116:32) {
status: 101,
signal: null,
output: [
null,
'',
"thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', src\\interface.rs:488:27\n" +
'note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n'
],
pid: 28304,
stdout: '',
stderr: "thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', src\\interface.rs:488:27\n" +
'note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n'
}
When fed to mathcat.exe built from MathCAT 0.1.23 with main.rs of:
// *** MathCAT doesn't normally want to build a binary ***
// *** This file is here because it is useful for trying out things ***
fn main() {
use libmathcat::interface::*;
use std::env;
let args: Vec<String> = env::args().collect();
let expr = &args[1];
let rules_dir = std::env::current_exe().unwrap().parent().unwrap().join("./Rules");
let rules_dir = rules_dir.as_os_str().to_str().unwrap().to_string();
if let Err(e) = set_rules_dir(rules_dir) {
panic!("Error: exiting -- {}", errors_to_string(&e)); }
if let Err(e) = set_mathml(expr.to_string()) {
panic!("Error: exiting -- {}", errors_to_string(&e));
};
set_preference("TTS".to_string(), "none".to_string()).unwrap();
set_preference("Impairment".to_string(), "Blindness".to_string()).unwrap();
set_preference("Language".to_string(), "en".to_string()).unwrap();
set_preference("SpeechSound".to_string(), "None".to_string()).unwrap();
set_preference("Verbosity".to_string(), "Verbose".to_string()).unwrap();
set_preference("MathRate".to_string(), "100".to_string()).unwrap();
set_preference("SpeechStyle".to_string(), "ClearSpeak".to_string()).unwrap();
set_preference("SubjectArea".to_string(), "General".to_string()).unwrap();
set_preference("Chemistry".to_string(), "SpellOut".to_string()).unwrap();
set_preference("BrailleCode".to_string(), "Nemeth".to_string()).unwrap();
let mut json_string = String::new();
json_string.push_str("{ version:'");
json_string.push_str(&get_version());
json_string.push_str("', clearspeak:'");
match get_spoken_text() {
Ok(speech) => json_string.push_str(&speech),
Err(_e) => json_string.push_str("ERROR"),
}
json_string.push_str("', braille:'");
set_preference("BrailleCode".to_string(), "Nemeth".to_string()).unwrap();
match get_braille("".to_string()) {
Ok(braille) => json_string.push_str(&braille),
Err(_e) => json_string.push_str("ERROR"),
}
json_string.push_str("'}");
println!("{}", json_string);
}
If I make a call to set_rules_dir with an invalid path, then call it again with a valid rules directory, calls to set_mathml will still give an error.
I can see a number of cases where this matters:
The Nemeth braille rendered by MathCAT from the MathML for these two aligned equations:
results in braille that has no separation between the '5' at the end of the first line and the '3.1x' at the start of the second line:
⠼⠦⠨⠲⠲⠭⠬⠢⠢⠽⠀⠨⠅⠀⠼⠢⠒⠨⠂⠭⠤⠴⠨⠶⠀⠨⠅⠀⠤⠼⠂⠨⠂
The braille is roughly equivalent to "8.44 x plus 55 y equals 53.1 x minus 0.7 equals negative 1.1"
Note the 53.1 x in the braille output instead of a separation between the two equations.
At a minimum, a space between the two equations is needed:
⠼⠦⠨⠲⠲⠭⠬⠢⠢⠽⠀⠨⠅⠀⠼⠢ ⠼⠒⠨⠂⠭⠤⠴⠨⠶⠀⠨⠅⠀⠤⠼⠂⠨⠂
Such multiline content is problematic. Unfortunately, it does not appear that Nemeth code has a mechanism for rendering a new line indicator. It would be worth researching if adding "Line 1: " ... "Line 2:", (or similar indication) would be appreciated.
The LaTeX was:
\begin{aligned}
8.44x + 55y &= 5\\
3.1x - 0.7 &= -1.1
\end{aligned}
With MSAA UIA settings as "Always", in Word, if you create an equation and then use INSERT + alt + m, you hear the MathML text read aloud (mn, etc). This doesn't happen with MathPlayer.
This didn't happen in beta 2, but does in beta 3.
To find crashes and other problems (not incorrect input), fuzz testing is useful. It looks like there are a few Rust fuzz testing cargos.
They need investigation. Some don't run on Windows and would need to be run on linux... which is easy to do with github actions.
When using MathCat 0.1.2 with an MS Word document + MathType equations, MathCat will not speak the math.
Instead, I hear "graphic MathType 6.0 equation".
E.g., The follow line from an MS Word document will read:
Remember, the derivative of a function "graphic MathType 6.0 equation" at a value "graphic MathType 6.0 equation" is "graphic MathType 6.0 equation".
Note: The version of MathType that I am using is actually 7.4.8.0 not 6.0
Hi,
I'm curious if MathCAT could be directly called by AT like the JAWS screen-reading software. JAWS is not reading MathML in a clear, unambiguous manner (I filed an issue on this here: FreedomScientific/standards-support#539 ).
Rather than AT providers trying (and failing) to do this work on their own, will MathCAT have an API so AT providers can call it instead of their being a universe full of varied mathematics to speech implementations? I would think that would foster consistency which would greatly improve the easy of comprehension.
Are their own going efforts to have MathCAT adopted by JAWS in particular? I'd like to supplement my bug report to Freedom Scientific with a suggestion to adopt MathCAT if that is appropriate.
Thanks for all the great work!
Example:
<math>
<mrow>
<mfrac>
<mn>1</mn>
<mrow>
<mi>a</mi><mi>c</mi></mrow>
</mfrac>
<msub>
<mrow>
<mi>log</mi></mrow>
<mn>2</mn>
</msub>
<mrow><mo>(</mo>
<mrow>
<mi>k</mi><mo>−</mo><mi>d</mi></mrow>
<mo>)</mo></mrow></mrow>
</math>
View the fraction 1 over ac, log base 2 …
Zoom in one level, and then move right so that “log base 2 (k-d)” is selected.
Note that the left highlight is still under the Close Fraction symbol, Dots 3-4-5-6, instead of the l in log, Dots 1-2-3.
The problem is that the "end fraction" braille symbol is the same as the "numeric indicator", so the code thinks it should be included.
Look at this, I see UEB is not handled at all although some things such as the numeric indicator and cap indicator are the same, so it kind of works. Typeface changes are different though.
The MathCAT Nemeth braille output for a matrix does not indicate the enlarged open and closing grouping symbols.
For example, when provided the XML for a 3x3 identity matrix:
The MathCAT braille output is equivalent to "(1 0 0)(0 1 0)(0 0 1)":
⠷⠂⠀⠼⠴⠀⠼⠴⠾⠷⠴⠀⠼⠂⠀⠼⠴⠾⠷⠴⠀⠼⠴⠀⠼⠂⠾
The MathCAT speech output is recognizing the matrix:
the 3 by 3 matrix; column 1; 1; column 2; 0; column 3; 0; column 1; 0; column 2; 1; column 3; 0; column 1; 0; column 2; 0; column 3; 1;
Expected braille output:
From: The APH Nemeth Tutorial Lesson 9.4: Matrices:
Enlarged brackets
The enlarged brackets are three-cell symbols. Dot four is used to indicate the bracket modification; dot six, the capitalization indicator, to signify enlargement; and the opening or closing sign of grouping.enlarged opening bracket, dot four dot six dots one two three five six
enlarged closing bracket, dot four dot six dots two three four five six
See also, Rule XVIII-[§126] top of page 127 of the 1972 Nemeth Braille Code pdf).
Thus, an expected Nemeth braille rendering for the above identity matrix could be:
⠈⠠⠷⠂⠀⠼⠴⠀⠼⠴⠈⠠⠾ ⠈⠠⠷⠴⠀⠼⠂⠀⠼⠴⠈⠠⠾ ⠈⠠⠷⠴⠀⠼⠴⠀⠼⠂⠈⠠⠾
To reproduce, here is the MathML (example from the MathML specification):
<math xmlns="http://www.w3.org/1998/Math/MathML">
<mrow>
<mo> ( </mo>
<mtable>
<mtr>
<mtd>
<mn>1</mn>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
</mtr>
<mtr>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mn>1</mn>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
</mtr>
<mtr>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mn>1</mn>
</mtd>
</mtr>
</mtable>
<mo> ) </mo>
</mrow>
</math>
It can also be recreated using this LaTeX:
$\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}$
Turn on for roots, fractions, and matrices.
The following fails to put the number together properly:
<math><mn>57</mn><mo>.</mo><mn>2</mn><mo> </mo><mi>cm</mi></math>
The result is:
<math>
<mrow data-changed='added'>
<mn>57</mn>
<mo>.</mo>
<mrow data-changed='added'>
<mn>2 </mn>
<mo data-changed='added'>⁢</mo>
<mi>cm</mi>
</mrow>
</mrow>
</math>
The problem appears to be the non-breaking space. Remove it, and the number is correct.
The file has a large number of private space entries that probably came from some old version of MathType where it couldn't write the plane 1 unicode values. I tried a somewhat old version of MathType and it is using mathvariant
. The private unicode ones should be flushed as they take up more space and more time to read them in.
Also, in the alphanumeric block, there are a lot of entries like
- "𝚊": [t: a] # 0x1d68a
and
- "𝙰": # 0x1d670
- t: ""
- test:
if: "($Blind or $Verbosity!='Terse')"
then: {t: cap}
- t: a
These should be changed to be more like the entries for the regular letters:
- "b-z":
- test:
if: "$TTS='none'"
then: [t: "."]
else: [spell: "."]
- "B-Z":
- test:
if: "$TTS='none'"
# note: processing of ranges converts '.' into the character, so it needs to be in quotes below
then: [x: "$CapitalLetters", x: "translate('.', 'BCDEFGHIJKLMNOPQRSTUVWXYZ', 'bcdefghijklmnopqrstuvwxyz')"]
else: {spell: "."}
These will take advantage of the speech engines appropriately. It will also shorten the file some, but probably not change read time much since the range needs to get expanded. Still probably faster than I/O.
It would be good to do these for the Greek letters also. However, I don't think ranges can be used because each Greek letter needs to be named. I need to see what the speech engines do if I give them a Greek letter and tell them to interpret-as='characters'
(the SSML command). If that works across all the speech engines (including Eloquence), then the range can be used.
The following fails to make the 2
be the base of the mmultiscripts:
<math> <msup> <mrow/> <mo>−</mo></msup> <mn>2</mn> </math>
Instead, it comes up with:
<math>
<mrow data-changed='added'>
<mmultiscripts>
<msup>
<mtext data-changed='empty_content'> </mtext>
<mo>-</mo>
</msup>
</mmultiscripts>
<mo data-changed='added'>⁢</mo>
<mn>2</mn>
</mrow>
</math>
There are a number of times where some semi-complicated test is repeated for various elements. E.g., for UEB the following occurs 6 times:
if: # FIX: need to add arbitrary shapes here (see also msub/msup)
- "*[2][self::m:mn or"
# next test is to make sure there is just one symbol
- " ((self::m:mi or self::m:mo) and string-length(text())=1) or"
- " self::m:mfrac or self::m:msqrt or self::m:mroot or"
- " IsBracketed(., '(', ')') or IsBracketed(., '[', ']') or IsBracketed(., '{', '}') or"
- " IsInDefinition(., 'Arrows')"
- " ]"
Actually, the child (*[2]
) varies, but the rest is the same.
Maybe something new like
-
name: IsItem
replace: ...
The missing match
clause signals this goes into the global environment for this rule set.
It would be used like other variables:
if: "$IsItem(*[2])"
where the *[2]
would not be part of the definition of IsItem
.
This is slightly limited in that only one parameter is allowed.
Note: this version needs to be evaluated at runtime. I think that currently when a variable is pushed on the stack, it is evaluated at that point. In this global case, it needs to wait until the function is called/referenced to do the evaluation.
For fun, I fed some content MathML to MathCat 0.1.23 and got an expected error from get_spoken_text()
but received an Ok result and gibberish braille instead of an expected error from get_braille("".to_string())
:
u⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tapplyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼teqe⠐ptyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tciu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼t⠐atrixu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼t⠐atrixrowe⠐ptyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tcie⠐ptyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tciu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼t⠐atrixrowe⠐ptyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tcie⠐ptyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tci
The MathML was:
<math xmlns="http://www.w3.org/1998/Math/MathML">
<mrow>
<mi>A</mi>
<mo>=</mo>
<mfenced open="[" close="]">
<mtable>
<mtr>
<mtd>
<mi>x</mi>
</mtd>
<mtd>
<mi>y</mi>
</mtd>
</mtr>
<mtr>
<mtd>
<mi>z</mi>
</mtd>
<mtd>
<mi>w</mi>
</mtd>
</mtr>
</mtable>
</mfenced>
</mrow>
</math>
This remains unimplemented.
This example comes from section 3.2.5.5 Invisible operators of the MathML3 spec.
<math>
<msub>
<mi> m </mi>
<mrow>
<mn> 1 </mn>
<mo> ⁣<!--INVISIBLE SEPARATOR--> </mo>
<mn> 2 </mn>
</mrow>
</msub>
</math>
The MathCAT ClearSpeak and SimpleSpeak both produce usable output that separates the two number values by a space:
m sub 1 2
However, the Nemeth braille output concatenates the number values thus rendering it as m sub twelve:
⠍⠰⠂⠆
The subscript-superscript comma (dots 246), always without a following space, is used to transcribe a comma in superscripts and subscripts (See Rule XIII-[§78] on page 88 of the 1972 Nemeth Braille Code pdf).
Thus, one could expect the following for the Nemeth braille:
⠍⠰⠂⠪⠆
Currently, the is_up_to_date
check can only check the initial file. The included files should also be checked.
To do this, when a file is included, it (a FileAndTime
) should be added to a Vec
stored in PreferenceManager
. One for the speech rules and one for Unicode rules. The is_up_to_date
check should go through that list in determining whether the file is up to date.
Another option is to change the current FileAndTime
struct to be a Vec
of FileAndTime
(shrunk to being just a single file). The Vec
is updated as include
s are processed. Currently the "time" part of FileAndTime
only allows for a single time so you can't know if some included file changed.
See #12 for how to handle the include chain.
There is currently a check to avoid speaking punctuation at the end of a math expression.
This Wikipedia page has a .
followed by a <mspace with='thinmathspace'/>
and that is not being caught.
Make canonicalization throw out spaces at the start/end (if in math
/mrow
).
For some double struck letters, like ⅇ (ExponentialE) and ⅆ (DifferentialD), the Nemeth braille output appropriately ignores the font styling:
comes out simply as ⠮⠰⠴⠘⠂⠐⠑⠘⠭⠐⠙⠭
.
<math>
<mrow>
<msubsup>
<mo> ∫<--int--> </mo>
<mn> 0 </mn>
<mn> 1 </mn>
</msubsup>
<mrow>
<msup>
<mi> ⅇ<--ExponentialE--> </mi>
<mi> x </mi>
</msup>
<mo> ⁢<--InvisibleTimes--> </mo>
<mrow>
<mo> ⅆ<--DifferentialD--> </mo>
<mi> x </mi>
</mrow>
</mrow>
</mrow>
</math>
For ⅈ (ImaginaryI), the Nemeth braille output appears to be trying to indicate the special symbol:
comes out as ⠼⠆⠬⠒ⅈ
(note the raw double struck italic i character in the braille output -- the braille display we tested rendered the double struck italic i as it's numeric unicode char value).
<math xmlns="http://www.w3.org/1998/Math/MathML">
<mrow>
<mn> 2 </mn>
<mo> + </mo>
<mrow>
<mn> 3 </mn>
<mo> ⁢<--InvisibleTimes--> </mo>
<mi> ⅈ<--ImaginaryI--> </mi>
</mrow>
</mrow>
</math>
A Wikipedia page (first mtr
, second mtd
of next to last entry) starts with an empty mi
. For some reason, the default intent.yaml rule dies with
Replacement
[test: [ if: 'x: "count(*) > 0"' then:[intent: xpath-name: x: "name(.)"
children: [x: "*"]] else:[intent: xpath-name: x: "name(.)"
children: [x: "text()"]]]
]
...due to matching the following MathML with the pattern
x: "."
<mi id='M283ygl4-17' data-id-added='true'></mi>
The patterns are in C:\Users\neils\MathCAT\Rules\intent.yaml.
caused by: replacing inside 'intent'
caused by: During replacement, no matching element found
Sample MathML from the MathML3 spec for generated both truncated speech output and erroneous Nemeth braille output.
<math>
<mmultiscripts>
<mi>R</mi>
<mi>i</mi>
<none></none>
<none></none>
<mi>j</mi>
<mi>k</mi>
<none></none>
<mi>l</mi>
<none></none>
</mmultiscripts>
</math>
Resulted in speech output of simply 'R' from the MathCAT demo and 'cap r' from MathCAT.
The Nemeth braille output received from MathCAT demo was (cap r subscript i):
⠠⠗⠰⠊
The Duxbury braille translator produced the following braille for the above example:
⠠⠗⠰⠊⠅⠇⠘⠚
The MathJax SRE 4.0 produced the following MathSpeak Grammar verbose output which seems to mirror the Duxbury braille:
upper R Subscript i k l Superscript j
I am not sure if the order of the subscripts and superscripts is important? Neither Duxbury's braille nor MathJax SRE's speech honored the sequence as rendered.
The following example is a simplified form found in https://en.wikipedia.org/wiki/Fundamental_theorem_of_algebra#Bounds_on_the_zeros_of_a_polynomial
<math xmlns='http://www.w3.org/1998/Math/MathML' >
<mrow>
<mrow class='MJX-TeXAtom-ORD'>
<mo stretchy='false'>|</mo>
</mrow>
<msub>
<mi>a</mi>
<mrow class='MJX-TeXAtom-ORD'>
<mi>k</mi>
</mrow>
</msub>
<msup>
<mrow class='MJX-TeXAtom-ORD'>
<mo stretchy='false'>|</mo>
</mrow>
<mrow class='MJX-TeXAtom-ORD'>
<mn>2</mn>
</mrow>
</msup>
</mrow>
</math>
Canonicalization will lift the |
s out of the mrows. The second |
is inside of an msup
. That msup
needs to the enclosing mrow
so that the base is the mrow
and the superscript is the 2
. This will then allow the rules to recognize that this is an absolute value.
mhchem/mathjax converts the arrows <=>
, <<=>
, and <=>>
and mover
with Unicode arrows because there is (currently) no Unicode equivalent. Here's what they look like:
See below for details of what the MathML looks like.
Probably the best thing to do is to handle this in canonicalization and convert them to mo
s with a private space Unicode spot. Or possibly anticipate them being added to the empty slots in the arrow block: 2B74, 2B75, and 2B96.
Then speech and braille needs to be added to the appropriate unicode.yaml files.
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
...
<mover>
<mrow data-mjx-texclass="OP">
<mrow data-mjx-texclass="ORD">
<mpadded height="0" depth="0">
<mrow data-mjx-texclass="ORD">
<mo stretchy="false">↽</mo>
</mrow>
<mstyle scriptlevel="0">
<mspace width="-0.167em"></mspace>
</mstyle>
<mstyle scriptlevel="0">
<mspace width="-0.167em"></mspace>
</mstyle>
<mrow data-mjx-texclass="ORD">
<mo>−</mo>
</mrow>
</mpadded>
</mrow>
</mrow>
<mrow data-mjx-texclass="ORD">
<mstyle displaystyle="false" scriptlevel="0">
<mrow data-mjx-texclass="ORD">
<mo>−</mo>
</mrow>
<mstyle scriptlevel="0">
<mspace width="-0.167em"></mspace>
</mstyle>
<mstyle scriptlevel="0">
<mspace width="-0.167em"></mspace>
</mstyle>
<mrow data-mjx-texclass="ORD">
<mo stretchy="false">⇀</mo>
</mrow>
</mstyle>
</mrow>
</mover>
....
</math>
which when you strip it down is (for <=>
:)
<mover>
<mrow>
<mo stretchy='false'>↽</mo>
<mo>-</mo>
</mrow>
<mrow>
<mo>-</mo>
<mo stretchy='false'>⇀</mo>
</mrow>
</mover>
for <<=>
:
<mover>
<mrow depth='0' height='0' data-changed='added'>
<mo stretchy='false'>↽</mo>
<mo>-</mo>
</mrow>
<mo displaystyle='false' scriptlevel='0' data-changed='added' stretchy='false'>⇀</mo>
</mover>
for <=>>
:
<mover>
<mo stretchy='false'>↽</mo>
<mrow displaystyle='false' scriptlevel='0' data-changed='added'>
<mo>-</mo>
<mo stretchy='false'>⇀</mo>
</mrow>
</mover>
Note the mrow
around the longer arrow.
In the early days, the plan was to have up to three locations/ancestors where a file could define things, going from most specific to more general to allow sharing. Later on, I added #include
to most rule file files.
The remaining case is the definition.yaml
files. Add it here.
The the array of three things and the needed lookup chaining can be eliminated.
From Dan Brown:
One thing I noticed with fractions, when using Down Arrow to zoom in, speech clearly announces “In numerator,” but doesn’t similarly announce “In denominator” when I press Right Arrow. Likewise, when I then switch back and forth on the same level from numerator to denominator with Left and Right Arrows, only the values are announced.
Ala MathCATForPython, create a MathCatForC project and write the interfaces.
For an alternate MathML rendering of
the Nemeth braille output leaves out the dot-5 multipurpose indicator before the opening parenthesis:
⠷⠩⠱⠹⠁⠌⠃⠼⠐⠾⠣⠱⠻
The expected Nemeth was:
⠐⠷⠩⠱⠻⠹⠁⠌⠃⠼⠐⠾⠣⠱⠻
The source MathML comes from 3.2.5.8 Stretching of operators, fences and accents in the MathML3 spec
<math xmlns="http://www.w3.org/1998/Math/MathML">
<mrow>
<munder>
<mo> ( </mo>
<mo> _<!--LOW LINE--> </mo>
</munder>
<mfrac>
<mi> a </mi>
<mi> b </mi>
</mfrac>
<mover>
<mo> ) </mo>
<mo> ‾<!--OVERLINE--> </mo>
</mover>
</mrow>
</math>
Also, not sure if this is an issue: The speech text output did not spell out the modifying characters: modified open paren with ¯ below, a over b, modified close paren with ‾ above
.
In a fraction, moving into the numerator/denominator should always say that it is in there. That's not happening all the time. In particularly, When moving at a char level, it doesn't announce it.
Need to implement options for ClearSpeak fractions
Repeatedly issuing "whereami" within some small period of time should walk up the tree. That is not currently implemented.
This should be based on an estimate of how long it takes to speak the command along some delay (10 seconds?). The delay should probably be a user preference.
Test case braille::UEB::iceb::hat_12_1_7
is:
The output uses a cap word indicator but the example doesn't. This is likely because the last part of the definition in section GTM 12 says "the item is simply the previous individual symbol". Symbol in this case includes the cap indicator. So it seems in this case,
The code in braille.rs
that builds a cap word from caps needs modification to handle "symbols".
MathCAT 0.1.23 when given the MathML for a variable with both a superscript and subscript generated unexpected ClearSpeak and SimpleSpeak output. For example,
Generated this ClearSpeak:
msub of, x comma k to the i-th power
and generated this SimpleSpeak:
msub of, x comma k to the i-th
.
We expected something more like:
x sub k raised to the i-th power
Here is the source MathML (generated by MathJax for x^{i}_{k}
):
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
<mstyle displaystyle="true" scriptlevel="0">
<msubsup>
<mi>x</mi>
<mrow data-mjx-texclass="ORD">
<mi>k</mi>
</mrow>
<mrow data-mjx-texclass="ORD">
<mi>i</mi>
</mrow>
</msubsup>
</mstyle>
</math>
The settings used were:
set_preference("TTS".to_string(), "none".to_string()).unwrap();
set_preference("Impairment".to_string(), "Blindness".to_string()).unwrap();
set_preference("Language".to_string(), "en".to_string()).unwrap();
set_preference("SpeechSound".to_string(), "None".to_string()).unwrap();
set_preference("Verbosity".to_string(), "Verbose".to_string()).unwrap();
set_preference("MathRate".to_string(), "100".to_string()).unwrap();
set_preference("SpeechStyle".to_string(), "ClearSpeak".to_string()).unwrap();
set_preference("SubjectArea".to_string(), "General".to_string()).unwrap();
set_preference("Chemistry".to_string(), "SpellOut".to_string()).unwrap();
set_preference("BrailleCode".to_string(), "Nemeth".to_string()).unwrap();
In this example just above https://en.wikipedia.org/wiki/Fundamental_theorem_of_algebra#Complex-analytic_proofs, the absolute value is not being spoken for the first and last instances. Instead, it is saying "divides".
<math xmlns='http://www.w3.org/1998/Math/MathML' alttext='{\displaystyle {\tfrac {1}{2}}|z^{n}|<|p(z)|<{\tfrac {3}{2}}|z^{n}|}'>
<semantics>
<mrow class='MJX-TeXAtom-ORD'>
<mstyle scriptlevel='0' displaystyle='true'>
<mrow class='MJX-TeXAtom-ORD'>
<mstyle scriptlevel='0' displaystyle='false'>
<mfrac>
<mn>1</mn>
<mn>2</mn>
</mfrac>
</mstyle>
</mrow>
<mrow class='MJX-TeXAtom-ORD'>
<mo stretchy='false'>|</mo>
</mrow>
<msup>
<mi>z</mi>
<mrow class='MJX-TeXAtom-ORD'>
<mi>n</mi>
</mrow>
</msup>
<mrow class='MJX-TeXAtom-ORD'>
<mo stretchy='false'>|</mo>
</mrow>
<mo><</mo>
<mrow class='MJX-TeXAtom-ORD'>
<mo stretchy='false'>|</mo>
</mrow>
<mi>p</mi>
<mo stretchy='false'>(</mo>
<mi>z</mi>
<mo stretchy='false'>)</mo>
<mrow class='MJX-TeXAtom-ORD'>
<mo stretchy='false'>|</mo>
</mrow>
<mo><</mo>
<mrow class='MJX-TeXAtom-ORD'>
<mstyle scriptlevel='0' displaystyle='false'>
<mfrac>
<mn>3</mn>
<mn>2</mn>
</mfrac>
</mstyle>
</mrow>
<mrow class='MJX-TeXAtom-ORD'>
<mo stretchy='false'>|</mo>
</mrow>
<msup>
<mi>z</mi>
<mrow class='MJX-TeXAtom-ORD'>
<mi>n</mi>
</mrow>
</msup>
<mrow class='MJX-TeXAtom-ORD'>
<mo stretchy='false'>|</mo>
</mrow>
</mstyle>
</mrow>
<annotation encoding='application/x-tex'>{\displaystyle {\tfrac {1}{2}}|z^{n}|<|p(z)|<{\tfrac {3}{2}}|z^{n}|}</annotation>
</semantics>
</math>
Neither the speech rules nor the navigation rules work if rowspan
or columnspan
are given on an mtable
element.
For navigation, it might be useful for the canonicalize step to mark row/col numbers and also id
s of the up/down/before/after elements since those are hard to compute with xpath.
Navigation may also want to add a state variable for current row/col since moving to a spanned row/col potentially loses the current row/col when you move to a non-spanned cell.
MathCAT, lacks support for elementary math in:
The speech and navigation can probably be ported over from MathPlayer.
The braille requires a solution for displaying 2D braille.
In the Navigation section we have Overview
and ResetOverView
.
Consider whether the case usage should be consistent.
In the prefs.yaml file, it appears to have incorrect guidance in the comments
The ClearSpeak setting on line 17 is listed as follows:
Roots: Auto # PosNegSqRoot, RootEnd, PosNegSqRootEnd
However, "RootEnd" has no effect. Whereas "EndRoot" seems to.
The three examples below generate the following error in MathCat Demo (using v0.1.17):
Pattern match/replacement failure!
caused by: attempting replacement pattern: "default" for "math".
Replacement
[test: [ if: 'x: "$MathRate = 100"' then:[x: "*"] else:[rate: 987654321.5[x: "*"]]]
]
...due to matching the following MathML with the pattern
x: "."
\begin{equation}
\begin{split}
A & = \frac{\pi r^2}{2} \\
& = \frac{1}{2} \pi r^2
\end{split}
\end{equation}
\begin{align}
\frac{d}{dx} \ln x &= \lim_{h\to 0} \frac{\ln(x+h) - \ln x}{h} \\
&= \ln e^{1/x}\\
&= \frac{1}{x}
\end{align}
\begin{aligned}
x &=\frac{a}{c} + \frac{b}{c} \\
&=\frac{a+b}{c}
\end{aligned}
\begin{align}
\frac{d}{dx} \ln x &= \lim_{h\to 0} \frac{\ln(x+h) - \ln x}{h} \\
\frac{d}{dx} \ln x &= \ln e^{1/x} && \text{How this follows is left as an exercise.}\\
\frac{d}{dx} \ln x &= \frac{1}{x} && \text{Using the definition of ln as inverse function}
\end{align}
If the columns each have content, the error goes away:
\begin{equation}
\begin{split}
A & = \frac{\pi r^2}{2} \\
A & = \frac{1}{2} \pi r^2
\end{split}
\end{equation}
\begin{align}
\frac{d}{dx} \ln x &= \lim_{h\to 0} \frac{\ln(x+h) - \ln x}{h} \\
\frac{d}{dx} \ln x &= \ln e^{1/x}\\
\frac{d}{dx} \ln x &= \frac{1}{x}
\end{align}
\begin{aligned}
x &=\frac{a}{c} + \frac{b}{c} \\
x &=\frac{a+b}{c}
\end{aligned}
\begin{align}
\frac{d}{dx} \ln x &= \lim_{h\to 0} \frac{\ln(x+h) - \ln x}{h} \\
\frac{d}{dx} \ln x &= \ln e^{1/x} & \text{dummy} & \text{How this follows is left as an exercise.}\\
\frac{d}{dx} \ln x &= \frac{1}{x} & \text{dummy} & \text{Using the definition of ln as inverse function}
\end{align}
For chemistry, the rule for chemical elements is:
- spell: text()
That works well most of the time. However, if the user pref SpeechOverrides_CapitalLetters
is set to a string such as cap
, then the word cap
gets spelled out also. When the tts is none
, that results in c a p
and when it is set to a value such ass ssml
, you get for S
<say-as interpret-as='characters'>cap <say-as interpret-as='characters'>s</say-as></say-as>
due to the unicode.yaml
rule
- "B-Z":
- test:
# note: processing of ranges converts '.' into the character, so it needs to be in quotes below
if: "$SpeechOverrides_CapitalLetters = ''"
then: [spell: "'.'"]
else: [x: "$SpeechOverrides_CapitalLetters", spell: "translate('.', 'BCDEFGHIJKLMNOPQRSTUVWXYZ', 'bcdefghijklmnopqrstuvwxyz')"]
Because the special rule for 'a' in English, you can't move implementation of $SpeechOverrides_CapitalLetters
into the implementation of spell
. Maybe add a check for "English" in the spell
implementation?
An alternative might be to add a no_spell
rule to turn off spelling.
The ClearSpeak output for math with a top or bottom brace includes the ASCII character for the symbol instead of spelling out the name of the symbol. For example, for:
MathCat 0.1.23 generated the following ClearSpeak:
modified x plus y plus z with ⏞ above
We expected something like:
modified x plus y plus z with top brace above
The MathML was:
<math xmlns="http://www.w3.org/1998/Math/MathML">
<mover>
<mrow>
<mi>x</mi>
<mo>+</mo>
<mi>y</mi>
<mo>+</mo>
<mi>z</mi>
</mrow>
<mo>⏞</mo>
</mover>
</math>
The bottom brace example:
generated the following ClearSpeak:
modified x plus y plus z with ⏟ below
We expected something like:
modified x plus y plus z with bottom brace below
The MathML was:
<math xmlns="http://www.w3.org/1998/Math/MathML">
<munder>
<mrow>
<mi>x</mi>
<mo>+</mo>
<mi>y</mi>
<mo>+</mo>
<mi>z</mi>
</mrow>
<mo>⏟</mo>
</munder>
</math>
Start 4ac
. Zoom in the '4'. Then move right to the a
. The G1 indicator (⠰) disappears. If you move right to the c
, it reappears.
To reproduce:
Using the NVDA MathCAT add-on, attempt to read a MathML expression that causes a crash.
NVDA emits an error tone, and leaves a Python traceback in the log.
Perhaps there's a separate issue for the add-on: would it be possible to log the error without causing a Python traceback (e.g., catch the error, log it and continue)? A major drawback of the current implementation is that the user can't even navigate past the problematic MathML expression to continue reading the document (e.g., with the Down-Arrow key).
The first example is taken from the Wikipedia article on the Fundamental Theorem of Calculus:
https://en.wikipedia.org/wiki/Fundamental_theorem_of_calculus
The second example appears in the Wikipedia entry on elementary algebra:
https://en.wikipedia.org/wiki/Elementary_algebra
Zip file containing instances of the MathML that crash and also the corresponding log files:
mathcat-crashes.zip
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.