nsoiffer/mathcat

MathCAT: Math Capable Assistive Technology for generating speech, braille, and navigation.

mathcat's Issues

Support for MathML DTD's named entity definitions?

Will MathCAT be adding support for the named entity definitions in the MathML DTD (⁡, ⅆ, ⅇ, ∬, ⁢, ±, etc.)?

I started creating test sets of MathML to look at the ClearSpeak and braille output compared to that produced by the SRE and noticed MathCat returns "XML parsing error at xx: {UnknownNamedReference}" when the provided MathML includes named entities.

For examples, please see:

• https://brichwin.pages.iu.edu/tests/tests0001.html#ex59
• https://brichwin.pages.iu.edu/tests/tests0001.html#ex60
• https://brichwin.pages.iu.edu/tests/tests0001.html#ex64
• https://brichwin.pages.iu.edu/tests/tests0001.html#ex65

Zoom out not saying "zoomed out all the way"

Title pretty much say it all. It just says "zoom out; speech for math". Sometimes it repeats "zoom out".

Idea for better "IsFunctionName" detection

For probability, an expression often repeats "P(...)". If the arg is simple, the canonicalization decides it is a function. But if it isn't (e.g., "P(A∩B)", then it comes up with "times".

Perhaps a more global search (in is_single_arg()) to see if any of the "P(...)" inside of math have a simple arg, and if so consider all of them to be function call. An optimization would be store a Vec of found function names in CanonicalizeContext so the more global search only happens once. The stored value would be "(name, bool)".

Another thing to do is to have a subject area specific list of probable function names. Maybe the current list has subject area "General" which is picked up by all (almost all?) subject areas.

set_mathml can alter MathML to be invalid

I have encountered a case where the call to set_mathml leads to it altering the MathML to invalid MathML. Should get_braille then be called it will lead to an error. Below is the MathML I pass to the set_mathml call:
<math xmlns="http://www.w3.org/1998/Math/MathML" alttext="mml&lt;mtable columnalign=&quot;left&quot; equalrows=&quot;true&quot; equalcolumns=&quot;true&quot;/&gt;(mml&lt;mtr columnalign=&quot;left&quot;/&gt;mml&lt;mtd columnalign=&quot;left&quot;/&gt;(bb I = bb text(Prt       ) bb text(Write the simple interest formula) .) mml&lt;mtr columnalign=&quot;left&quot;/&gt;mml&lt;mtd columnalign=&quot;left&quot;/&gt;(text(    ) = bb 500 (bb 0.03) ⁡ (bb 3) text(    ) bb text(Substitute ) bb 500 bb text( for ) bb P text( )_, bb 0.03 bb text( for ) bb r , bb text( and ) bb 3 bb text( for ) bb t .) mml&lt;mtr columnalign=&quot;left&quot;/&gt;mml&lt;mtd columnalign=&quot;left&quot;/&gt;(text(  ) = 45 text(        ) bb text(Multiply) .))"><mtable columnalign="left" equalcolumns="true" equalrows="true"><mtr columnalign="left"><mtd columnalign="left"><mstyle mathvariant="bold"><mo> </mo></mstyle><mi>I</mi><mo> </mo><mo>=</mo><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext>Prt       </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext>Write the simple interest formula</mtext></mrow><mo> </mo><mo>.</mo></mtd></mtr><mo> </mo><mtr columnalign="left"><mtd columnalign="left"><mrow><mtext>    </mtext></mrow><mo> </mo><mo>=</mo><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>500</mn><mo> </mo><mrow><mo>(</mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>0.03</mn><mo>)</mo></mrow><mo> </mo><mo>⁡</mo><mo> </mo><mrow><mo>(</mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>3</mn><mo>)</mo></mrow><mo> </mo><mrow><mtext>    </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext>Substitute </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>500</mn><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext> for </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mi>P</mi><mo> </mo><msub><mrow><mtext> </mtext></mrow><mo>,</mo></msub><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>0.03</mn><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext> for </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mi>r</mi><mo> </mo><mo>,</mo><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext> and </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mn>3</mn><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext> for </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mi>t</mi><mo> </mo><mo>.</mo></mtd></mtr><mo> </mo><mtr columnalign="left"><mtd columnalign="left"><mrow><mtext>  </mtext></mrow><mo> </mo><mo>=</mo><mo> </mo><mn>45</mn><mo> </mo><mrow><mtext>        </mtext></mrow><mo> </mo><mstyle mathvariant="bold"><mo> </mo></mstyle><mrow><mtext>Multiply</mtext></mrow><mo> </mo><mo>.</mo></mtd></mtr></mtable></math>
In the altered MathML which is returned the msub only has a single child element when two are required.
I tried to create a minimal example <math><msub><mtext>&#x2009;</mtext><mo>,</mo></msub></math> this though is correctly handled with the msub being converted to a prescript. So I am still to find the minimal example to cause this issue.

Crash while reading a vector space-related expression.

The braille is rendered correctly, but MathCAT apparently crashes while producing the speech output.

The example is taken from
vector-example.zip
https://en.wikipedia.org/wiki/Vector_space

Need to add three arrows to Nemeth/UEB translations

In fixing #60, I made them be some (new) arrow code points.

These need to be handled for Nemeth and UEB translations.

There are some similar arrows, so they maybe get coded the same. Otherwise, they need to be dealt with as "arrow over arrow" constructs.

Unexpected Nemeth output for horizontal bar in five-step rule for directly-over/under modified expressions

The Nemeth output for directly-over and directly-under expressions modified by a horizonal bar results in "dots-36 dots-36 dots-36 dots-36" instead of simply using the Nemeth cell for horizonal bar: dots-156 (Rule XIV - Modifiers near bottom of page 98). See examples in Rule XIV Section 86 a. The Five-Step Rule for Transcribing Modified Expressions pages 99-101; Partially shown here:

For the expression:

MathCat produced this Nemeth:
⠐⠷⠩⠤⠤⠤⠤⠀⠻⠹⠁⠌⠃⠼⠐⠾⠣⠀⠤⠤⠤⠤⠀⠻
The expected Nemeth was:
⠐⠷⠩⠱⠻⠹⠁⠌⠃⠼⠐⠾⠣⠱⠻ (also note no spaces between the directly-over indicator ⠣ or directly-under indicator ⠩, the horizontal bar modifier ⠱, and the termination indicator ⠻)

LaTeX and MathML Code for the above example

\underline{(}\frac{a}{b}\overline{)}

<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
  <mstyle displaystyle="true" scriptlevel="0">
    <munder>
      <mo stretchy="false">(</mo>
      <mo accent="true">&#x2015;</mo>
    </munder>
    <mfrac>
      <mi>a</mi>
      <mi>b</mi>
    </mfrac>
    <mover>
      <mo stretchy="false">)</mo>
      <mo accent="true">&#x2015;</mo>
    </mover>
  </mstyle>
</math>

'index out of bounds' panic in interface.rs MathCAT 0.1.23

Source MathML

The following MathML generated an 'index out of bounds' panic in interface.rs:

<math xmlns="http://www.w3.org/1998/Math/MathML">
  <mrow>
    <mi>
      <mglyph fontfamily="my-braid-font" index="2" alt="23braid" />
    </mi>
    <mo>+</mo>
    <mi>
      <mglyph fontfamily="my-braid-font" index="5" alt="132braid" />
    </mi>
    <mo>=</mo>
    <mi>
      <mglyph fontfamily="my-braid-font" index="3" alt="13braid" />
    </mi>
  </mrow>
</math>

Error Received:

thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', src\interface.rs:488:27
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: Command failed: mathcat.exe <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi><mglyph fontfamily="my-braid-font" index="2" alt="23braid"/></mi><mo>+</mo><mi><mglyph fontfamily="my-braid-font" index="5" alt="132braid"/></mi><mo>=</mo><mi><mglyph fontfamily="my-braid-font" index="3" alt="13braid"/></mi></mrow></math>
thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', src\interface.rs:488:27
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

    at checkExecSyncError (node:child_process:828:11)
    at execFileSync (node:child_process:863:15)
    at callMathCAT (C:\Users\richw\dev\brailleMathTests\main2-tests0001b-0.1.23.js:78:24)
    at getMathResults (C:\Users\richw\dev\brailleMathTests\main2-tests0001b-0.1.23.js:116:32) {
  status: 101,
  signal: null,
  output: [
    null,
    '',
    "thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', src\\interface.rs:488:27\n" +
      'note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n'
  ],
  pid: 28304,
  stdout: '',
  stderr: "thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', src\\interface.rs:488:27\n" +
    'note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n'
}

Generating program

When fed to mathcat.exe built from MathCAT 0.1.23 with main.rs of:

// *** MathCAT doesn't normally want to build a binary ***
// *** This file is here because it is useful for trying out things ***

fn main() {
  use libmathcat::interface::*;
  use std::env;

  let args: Vec<String> = env::args().collect();
  let expr = &args[1];

  let rules_dir = std::env::current_exe().unwrap().parent().unwrap().join("./Rules");
  let rules_dir = rules_dir.as_os_str().to_str().unwrap().to_string();
  if let Err(e) = set_rules_dir(rules_dir) {
    panic!("Error: exiting -- {}", errors_to_string(&e));  }
  if let Err(e) = set_mathml(expr.to_string()) {
    panic!("Error: exiting -- {}", errors_to_string(&e));
  };

  set_preference("TTS".to_string(), "none".to_string()).unwrap();
  set_preference("Impairment".to_string(), "Blindness".to_string()).unwrap();
  set_preference("Language".to_string(), "en".to_string()).unwrap();
  set_preference("SpeechSound".to_string(), "None".to_string()).unwrap();
  set_preference("Verbosity".to_string(), "Verbose".to_string()).unwrap();
  set_preference("MathRate".to_string(), "100".to_string()).unwrap();
  set_preference("SpeechStyle".to_string(), "ClearSpeak".to_string()).unwrap();
  set_preference("SubjectArea".to_string(), "General".to_string()).unwrap();
  set_preference("Chemistry".to_string(), "SpellOut".to_string()).unwrap();
  set_preference("BrailleCode".to_string(), "Nemeth".to_string()).unwrap();

  let mut json_string = String::new();

  json_string.push_str("{ version:'");
  json_string.push_str(&get_version());

  json_string.push_str("', clearspeak:'");

  match get_spoken_text() {
    Ok(speech) => json_string.push_str(&speech),
    Err(_e) => json_string.push_str("ERROR"),
  }

  json_string.push_str("', braille:'");

  set_preference("BrailleCode".to_string(), "Nemeth".to_string()).unwrap();
  match get_braille("".to_string()) {
    Ok(braille) => json_string.push_str(&braille),
    Err(_e) => json_string.push_str("ERROR"),
  }
  json_string.push_str("'}");
  println!("{}", json_string);
}

Once invalid rules dir set it is not possible to recover by setting a valid rules dir

If I make a call to set_rules_dir with an invalid path, then call it again with a valid rules directory, calls to set_mathml will still give an error.
I can see a number of cases where this matters:

• Should the application let the user pick a rules directory, it would be tricky for the application to recover when the user goes on to select a valid rules directory. In some application work flows (eg. GUI applications) there should not be the need to restart just because of a change like this in a user setting.
• In testing bindings the library may be loaded once and shared amongst all tests. Should a test check that calling set_rules_dir will lead to the bindings throwing an appropriate Exception/error, this issue means no following tests will work. This is how I discovered this issue.

No separation in Nemeth braille output between content of neighboring lines in multiline aligned equations

The Nemeth braille rendered by MathCAT from the MathML for these two aligned equations:

results in braille that has no separation between the '5' at the end of the first line and the '3.1x' at the start of the second line:

⠼⠦⠨⠲⠲⠭⠬⠢⠢⠽⠀⠨⠅⠀⠼⠢⠒⠨⠂⠭⠤⠴⠨⠶⠀⠨⠅⠀⠤⠼⠂⠨⠂

The braille is roughly equivalent to "8.44 x plus 55 y equals 53.1 x minus 0.7 equals negative 1.1"

Note the 53.1 x in the braille output instead of a separation between the two equations.

Expected output

At a minimum, a space between the two equations is needed:

⠼⠦⠨⠲⠲⠭⠬⠢⠢⠽⠀⠨⠅⠀⠼⠢ ⠼⠒⠨⠂⠭⠤⠴⠨⠶⠀⠨⠅⠀⠤⠼⠂⠨⠂

Such multiline content is problematic. Unfortunately, it does not appear that Nemeth code has a mechanism for rendering a new line indicator. It would be worth researching if adding "Line 1: " ... "Line 2:", (or similar indication) would be appreciated.

To reproduce

The LaTeX was:

  \begin{aligned}
    8.44x + 55y &= 5\\
    3.1x - 0.7 &= -1.1
  \end{aligned}

NVDA 2022 beta 3 has problems with MathCAT in Word

With MSAA UIA settings as "Always", in Word, if you create an equation and then use INSERT + alt + m, you hear the MathML text read aloud (mn, etc). This doesn't happen with MathPlayer.

This didn't happen in beta 2, but does in beta 3.

Add fuzz testing to MathCAT

To find crashes and other problems (not incorrect input), fuzz testing is useful. It looks like there are a few Rust fuzz testing cargos.

They need investigation. Some don't run on Windows and would need to be run on linux... which is easy to do with github actions.

MathCat Not Reading MathType Equations in MS Word

When using MathCat 0.1.2 with an MS Word document + MathType equations, MathCat will not speak the math.

Instead, I hear "graphic MathType 6.0 equation".

E.g., The follow line from an MS Word document will read:

Remember, the derivative of a function "graphic MathType 6.0 equation" at a value "graphic MathType 6.0 equation" is "graphic MathType 6.0 equation".

Note: The version of MathType that I am using is actually 7.4.8.0 not 6.0

Question: Is MathCAT meant to be adopted by AT tools (like NVDA, JAWS, etc.)

Hi,

I'm curious if MathCAT could be directly called by AT like the JAWS screen-reading software. JAWS is not reading MathML in a clear, unambiguous manner (I filed an issue on this here: FreedomScientific/standards-support#539 ).

Rather than AT providers trying (and failing) to do this work on their own, will MathCAT have an API so AT providers can call it instead of their being a universe full of varied mathematics to speech implementations? I would think that would foster consistency which would greatly improve the easy of comprehension.

Are their own going efforts to have MathCAT adopted by JAWS in particular? I'd like to supplement my bug report to Freedom Scientific with a suggestion to adopt MathCAT if that is appropriate.

Thanks for all the great work!

Braille navigation highlights too much

Example:

<math>
      <mrow>
      <mfrac>
        <mn>1</mn>
        <mrow>
        <mi>a</mi><mi>c</mi></mrow>
      </mfrac>
      <msub>
        <mrow>
        <mi>log</mi></mrow>
        <mn>2</mn>
      </msub>
      <mrow><mo>(</mo>
        <mrow>
        <mi>k</mi><mo>−</mo><mi>d</mi></mrow>
      <mo>)</mo></mrow></mrow>
</math>

View the fraction 1 over ac, log base 2 …
Zoom in one level, and then move right so that “log base 2 (k-d)” is selected.
Note that the left highlight is still under the Close Fraction symbol, Dots 3-4-5-6, instead of the l in log, Dots 1-2-3.

The problem is that the "end fraction" braille symbol is the same as the "numeric indicator", so the code thinks it should be included.

Look at this, I see UEB is not handled at all although some things such as the numeric indicator and cap indicator are the same, so it kind of works. Typeface changes are different though.

Enlarged opening and closing matrix grouping symbols not indicated in Nemeth braille output.

The MathCAT Nemeth braille output for a matrix does not indicate the enlarged open and closing grouping symbols.

For example, when provided the XML for a 3x3 identity matrix:

The MathCAT braille output is equivalent to "(1 0 0)(0 1 0)(0 0 1)":

⠷⠂⠀⠼⠴⠀⠼⠴⠾⠷⠴⠀⠼⠂⠀⠼⠴⠾⠷⠴⠀⠼⠴⠀⠼⠂⠾

The MathCAT speech output is recognizing the matrix:

the 3 by 3 matrix; column 1; 1; column 2; 0; column 3; 0; column 1; 0; column 2; 1; column 3; 0; column 1; 0; column 2; 0; column 3; 1;

Expected braille output:
From: The APH Nemeth Tutorial Lesson 9.4: Matrices:

Enlarged brackets
The enlarged brackets are three-cell symbols. Dot four is used to indicate the bracket modification; dot six, the capitalization indicator, to signify enlargement; and the opening or closing sign of grouping.

enlarged opening bracket, dot four dot six dots one two three five six
enlarged closing bracket, dot four dot six dots two three four five six

See also, Rule XVIII-[§126] top of page 127 of the 1972 Nemeth Braille Code pdf).

Thus, an expected Nemeth braille rendering for the above identity matrix could be:

⠈⠠⠷⠂⠀⠼⠴⠀⠼⠴⠈⠠⠾ ⠈⠠⠷⠴⠀⠼⠂⠀⠼⠴⠈⠠⠾ ⠈⠠⠷⠴⠀⠼⠴⠀⠼⠂⠈⠠⠾

To reproduce, here is the MathML (example from the MathML specification):

<math xmlns="http://www.w3.org/1998/Math/MathML">
  <mrow>
    <mo> ( </mo>
    <mtable>
      <mtr>
        <mtd>
          <mn>1</mn>
        </mtd>
        <mtd>
          <mn>0</mn>
        </mtd>
        <mtd>
          <mn>0</mn>
        </mtd>
      </mtr>
      <mtr>
        <mtd>
          <mn>0</mn>
        </mtd>
        <mtd>
          <mn>1</mn>
        </mtd>
        <mtd>
          <mn>0</mn>
        </mtd>
      </mtr>
      <mtr>
        <mtd>
          <mn>0</mn>
        </mtd>
        <mtd>
          <mn>0</mn>
        </mtd>
        <mtd>
          <mn>1</mn>
        </mtd>
      </mtr>
    </mtable>
    <mo> ) </mo>
  </mrow>
</math>

It can also be recreated using this LaTeX:

$\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}$

ClearSpeak verbosity should turn on some disambiguating options

Turn on for roots, fractions, and matrices.

Number canonicalization failure

The following fails to put the number together properly:

<math><mn>57</mn><mo>.</mo><mn>2</mn><mo>&#xA0;</mo><mi>cm</mi></math>

The result is:

<math>
  <mrow data-changed='added'>
    <mn>57</mn>
    <mo>.</mo>
    <mrow data-changed='added'>
      <mn>2 </mn>
      <mo data-changed='added'>&#x2062;</mo>
      <mi>cm</mi>
    </mrow>
  </mrow>
 </math>

The problem appears to be the non-breaking space. Remove it, and the number is correct.

improve unicode-full.yaml

The file has a large number of private space entries that probably came from some old version of MathType where it couldn't write the plane 1 unicode values. I tried a somewhat old version of MathType and it is using mathvariant. The private unicode ones should be flushed as they take up more space and more time to read them in.

Also, in the alphanumeric block, there are a lot of entries like

- "𝚊": [t: a]                                   # 0x1d68a

and

 - "𝙰":                                          # 0x1d670
     - t: ""
     - test: 
         if: "($Blind or $Verbosity!='Terse')"
         then: {t: cap}
     - t: a

These should be changed to be more like the entries for the regular letters:

 - "b-z": 
    - test: 
        if: "$TTS='none'"
        then: [t: "."]                          
        else: [spell: "."]                       

 - "B-Z": 
    - test: 
        if: "$TTS='none'"
        # note: processing of ranges converts '.' into the character, so it needs to be in quotes below
        then: [x: "$CapitalLetters", x: "translate('.', 'BCDEFGHIJKLMNOPQRSTUVWXYZ', 'bcdefghijklmnopqrstuvwxyz')"]                         
        else: {spell: "."}            

These will take advantage of the speech engines appropriately. It will also shorten the file some, but probably not change read time much since the range needs to get expanded. Still probably faster than I/O.

It would be good to do these for the Greek letters also. However, I don't think ranges can be used because each Greek letter needs to be named. I need to see what the speech engines do if I give them a Greek letter and tell them to interpret-as='characters' (the SSML command). If that works across all the speech engines (including Eloquence), then the range can be used.

Bad/missed canonicalization to mmultiscripts

The following fails to make the 2 be the base of the mmultiscripts:

<math> <msup> <mrow/> <mo>&#x2212;</mo></msup> <mn>2</mn> </math>

Instead, it comes up with:

 <math>
  <mrow data-changed='added'>
    <mmultiscripts>
      <msup>
        <mtext data-changed='empty_content'> </mtext>
        <mo>-</mo>
      </msup>
    </mmultiscripts>
    <mo data-changed='added'>&#x2062;</mo>
    <mn>2</mn>
  </mrow>
 </math>

Add global variable/function definitions

There are a number of times where some semi-complicated test is repeated for various elements. E.g., for UEB the following occurs 6 times:

      if:   # FIX: need to add arbitrary shapes here (see also msub/msup)
      - "*[2][self::m:mn or"
              # next test is to make sure there is just one symbol
      - "     ((self::m:mi or self::m:mo) and string-length(text())=1) or"
      - "     self::m:mfrac or self::m:msqrt or self::m:mroot or"
      - "     IsBracketed(., '(', ')') or IsBracketed(., '[', ']') or IsBracketed(., '{', '}') or"
      - "     IsInDefinition(., 'Arrows')"
      - "    ]"

Actually, the child (*[2]) varies, but the rest is the same.

Maybe something new like

-
   name: IsItem
   replace: ...

The missing match clause signals this goes into the global environment for this rule set.

It would be used like other variables:

  if: "$IsItem(*[2])" 

where the *[2] would not be part of the definition of IsItem.

This is slightly limited in that only one parameter is allowed.

Note: this version needs to be evaluated at runtime. I think that currently when a variable is pushed on the stack, it is evaluated at that point. In this global case, it needs to wait until the function is called/referenced to do the evaluation.

Unexpected braille Ok result and gibberish braille output when expecting an error

For fun, I fed some content MathML to MathCat 0.1.23 and got an expected error from get_spoken_text() but received an Ok result and gibberish braille instead of an expected error from get_braille("".to_string()):

u⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tapplyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼teqe⠐ptyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tciu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼t⠐atrixu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼t⠐atrixrowe⠐ptyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tcie⠐ptyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tciu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼t⠐atrixrowe⠐ptyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tcie⠐ptyu⠼k⠼ow⠼⠐ath⠐lele⠐e⠼tci

The MathML was:

<math xmlns="http://www.w3.org/1998/Math/MathML">
  <mrow>
    <mi>A</mi>
    <mo>=</mo>
    <mfenced open="[" close="]">
      <mtable>
        <mtr>
          <mtd>
            <mi>x</mi>
          </mtd>
          <mtd>
            <mi>y</mi>
          </mtd>
        </mtr>
        <mtr>
          <mtd>
            <mi>z</mi>
          </mtd>
          <mtd>
            <mi>w</mi>
          </mtd>
        </mtr>
      </mtable>
    </mfenced>
  </mrow>
</math>

Need to implement navigation in tables

This remains unimplemented.

<mn> number values unexpectedly concatenated in Nemeth braille output from subscripts with invisible separators

This example comes from section 3.2.5.5 Invisible operators of the MathML3 spec.

<math>
  <msub>
    <mi> m </mi>
    <mrow>
      <mn> 1 </mn>
      <mo> &#x2063;<!--INVISIBLE SEPARATOR--> </mo>
      <mn> 2 </mn>
    </mrow>
  </msub>
</math>

The MathCAT ClearSpeak and SimpleSpeak both produce usable output that separates the two number values by a space:

m sub 1 2

However, the Nemeth braille output concatenates the number values thus rendering it as m sub twelve:

⠍⠰⠂⠆

The subscript-superscript comma (dots 246), always without a following space, is used to transcribe a comma in superscripts and subscripts (See Rule XIII-[§78] on page 88 of the 1972 Nemeth Braille Code pdf).

Thus, one could expect the following for the Nemeth braille:

⠍⠰⠂⠪⠆

Add include files to SpeechRules

Currently, the is_up_to_date check can only check the initial file. The included files should also be checked.

To do this, when a file is included, it (a FileAndTime) should be added to a Vec stored in PreferenceManager. One for the speech rules and one for Unicode rules. The is_up_to_date check should go through that list in determining whether the file is up to date.

Another option is to change the current FileAndTime struct to be a Vec of FileAndTime (shrunk to being just a single file). The Vec is updated as includes are processed. Currently the "time" part of FileAndTime only allows for a single time so you can't know if some included file changed.

See #12 for how to handle the include chain.

Add to check for ending punctuation

There is currently a check to avoid speaking punctuation at the end of a math expression.

This Wikipedia page has a . followed by a <mspace with='thinmathspace'/> and that is not being caught.

Make canonicalization throw out spaces at the start/end (if in math/mrow).

Nemeth - Inconsistent handling of double struck letters

For some double struck letters, like &#x2147; (ExponentialE) and &#x2146; (DifferentialD), the Nemeth braille output appropriately ignores the font styling:

comes out simply as ⠮⠰⠴⠘⠂⠐⠑⠘⠭⠐⠙⠭.

Click for the above integral's MathML source

<math>
  <mrow>
    <msubsup>
      <mo> &#x222B;<--int--> </mo>
      <mn> 0 </mn>
      <mn> 1 </mn>
    </msubsup>
    <mrow>
      <msup>
        <mi> &#x2147;<--ExponentialE--> </mi>
        <mi> x </mi>
      </msup>
      <mo> &#x2062;<--InvisibleTimes--> </mo>
      <mrow>
        <mo> &#x2146;<--DifferentialD--> </mo>
        <mi> x </mi>
      </mrow>
    </mrow>
  </mrow>
</math>

For &#x2148; (ImaginaryI), the Nemeth braille output appears to be trying to indicate the special symbol:

comes out as ⠼⠆⠬⠒ⅈ (note the raw double struck italic i character in the braille output -- the braille display we tested rendered the double struck italic i as it's numeric unicode char value).

Click for the above imaginary number's MathML source

<math xmlns="http://www.w3.org/1998/Math/MathML">
<mrow>
  <mn> 2 </mn>
  <mo> + </mo>
  <mrow>
    <mn> 3 </mn>
    <mo> &#x2062;<--InvisibleTimes--> </mo>
    <mi> &#x2148;<--ImaginaryI--> </mi>
  </mrow>
</mrow>
</math>

Empty mi causing failure

A Wikipedia page (first mtr, second mtd of next to last entry) starts with an empty mi. For some reason, the default intent.yaml rule dies with

Replacement
[test: [  if: 'x: "count(*) > 0"' then:[intent:  xpath-name: x: "name(.)"
      children: [x: "*"]] else:[intent:  xpath-name: x: "name(.)"
      children: [x: "text()"]]]
]
...due to matching the following MathML with the pattern
x: "."
 <mi id='M283ygl4-17' data-id-added='true'></mi>
The patterns are in C:\Users\neils\MathCAT\Rules\intent.yaml.

caused by: replacing inside 'intent'
caused by: During replacement, no matching element found

Erroneous truncated output from MathML with <mmultiscripts>

Sample MathML from the MathML3 spec for generated both truncated speech output and erroneous Nemeth braille output.

The MathML for:

 <math>
  <mmultiscripts>
    <mi>R</mi>
    <mi>i</mi>
    <none></none>
    <none></none>
    <mi>j</mi>
    <mi>k</mi>
    <none></none>
    <mi>l</mi>
    <none></none>
  </mmultiscripts>
 </math>

Resulted in speech output of simply 'R' from the MathCAT demo and 'cap r' from MathCAT.

The Nemeth braille output received from MathCAT demo was (cap r subscript i):

⠠⠗⠰⠊

Expected Output

The Duxbury braille translator produced the following braille for the above example:

⠠⠗⠰⠊⠅⠇⠘⠚

The MathJax SRE 4.0 produced the following MathSpeak Grammar verbose output which seems to mirror the Duxbury braille:

upper R Subscript i k l Superscript j

I am not sure if the order of the subscripts and superscripts is important? Neither Duxbury's braille nor MathJax SRE's speech honored the sequence as rendered.

Need canonicalization for script attached to close fence

The following example is a simplified form found in https://en.wikipedia.org/wiki/Fundamental_theorem_of_algebra#Bounds_on_the_zeros_of_a_polynomial

<math xmlns='http://www.w3.org/1998/Math/MathML' >
    <mrow>
          <mrow class='MJX-TeXAtom-ORD'>
            <mo stretchy='false'>|</mo>
          </mrow>
          <msub>
            <mi>a</mi>
            <mrow class='MJX-TeXAtom-ORD'>
              <mi>k</mi>
            </mrow>
          </msub>
          <msup>
            <mrow class='MJX-TeXAtom-ORD'>
              <mo stretchy='false'>|</mo>
            </mrow>
            <mrow class='MJX-TeXAtom-ORD'>
              <mn>2</mn>
            </mrow>
          </msup>
    </mrow>
</math>

Canonicalization will lift the |s out of the mrows. The second | is inside of an msup. That msup needs to the enclosing mrow so that the base is the mrow and the superscript is the 2. This will then allow the rules to recognize that this is an absolute value.

Handle three built-up arrows from mhchem

Issue

mhchem/mathjax converts the arrows <=>, <<=>, and <=>> and mover with Unicode arrows because there is (currently) no Unicode equivalent. Here's what they look like:

See below for details of what the MathML looks like.

Probably the best thing to do is to handle this in canonicalization and convert them to mos with a private space Unicode spot. Or possibly anticipate them being added to the empty slots in the arrow block: 2B74, 2B75, and 2B96.

Then speech and braille needs to be added to the appropriate unicode.yaml files.

Details

<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
         ...
        <mover>
          <mrow data-mjx-texclass="OP">
            <mrow data-mjx-texclass="ORD">
              <mpadded height="0" depth="0">
                <mrow data-mjx-texclass="ORD">
                  <mo stretchy="false">&#x21BD;</mo>
                </mrow>
                <mstyle scriptlevel="0">
                  <mspace width="-0.167em"></mspace>
                </mstyle>
                <mstyle scriptlevel="0">
                  <mspace width="-0.167em"></mspace>
                </mstyle>
                <mrow data-mjx-texclass="ORD">
                  <mo>&#x2212;</mo>
                </mrow>
              </mpadded>
            </mrow>
          </mrow>
          <mrow data-mjx-texclass="ORD">
            <mstyle displaystyle="false" scriptlevel="0">
              <mrow data-mjx-texclass="ORD">
                <mo>&#x2212;</mo>
              </mrow>
              <mstyle scriptlevel="0">
                <mspace width="-0.167em"></mspace>
              </mstyle>
              <mstyle scriptlevel="0">
                <mspace width="-0.167em"></mspace>
              </mstyle>
              <mrow data-mjx-texclass="ORD">
                <mo stretchy="false">&#x21C0;</mo>
              </mrow>
            </mstyle>
          </mrow>
        </mover>
        ....
</math>

which when you strip it down is (for <=>:)

    <mover>
      <mrow>
        <mo stretchy='false'>↽</mo>
        <mo>-</mo>
      </mrow>
      <mrow>
        <mo>-</mo>
        <mo stretchy='false'>⇀</mo>
      </mrow>
    </mover>

for <<=>:

   <mover>
      <mrow depth='0' height='0' data-changed='added'>
        <mo stretchy='false'>↽</mo>
        <mo>-</mo>
      </mrow>
      <mo displaystyle='false' scriptlevel='0' data-changed='added' stretchy='false'>⇀</mo>
    </mover>

for <=>>:

    <mover>
      <mo stretchy='false'>↽</mo>
      <mrow displaystyle='false' scriptlevel='0' data-changed='added'>
        <mo>-</mo>
        <mo stretchy='false'>⇀</mo>
      </mrow>
    </mover>

Note the mrow around the longer arrow.

Add #include capability to definition.yaml files

In the early days, the plan was to have up to three locations/ancestors where a file could define things, going from most specific to more general to allow sharing. Later on, I added #include to most rule file files.

The remaining case is the definition.yaml files. Add it here.

The the array of three things and the needed lookup chaining can be eliminated.

Navigation not speaking entry/exit from 2D structures in some cases

From Dan Brown:

One thing I noticed with fractions, when using Down Arrow to zoom in, speech clearly announces “In numerator,” but doesn’t similarly announce “In denominator” when I press Right Arrow. Likewise, when I then switch back and forth on the same level from numerator to denominator with Left and Right Arrows, only the values are announced.

Add a C/C++ interface to MathCAT

Ala MathCATForPython, create a MathCatForC project and write the interfaces.

Malformed Five-Step Rule for Transcribing Expressions Modified by Low Line and Overline Characters

For an alternate MathML rendering of

the Nemeth braille output leaves out the dot-5 multipurpose indicator before the opening parenthesis:
⠷⠩⠱⠹⠁⠌⠃⠼⠐⠾⠣⠱⠻

The expected Nemeth was:
⠐⠷⠩⠱⠻⠹⠁⠌⠃⠼⠐⠾⠣⠱⠻

The source MathML comes from 3.2.5.8 Stretching of operators, fences and accents in the MathML3 spec

<math xmlns="http://www.w3.org/1998/Math/MathML">
  <mrow>
    <munder>
      <mo> ( </mo>
      <mo> &#x5F;<!--LOW LINE--> </mo>
    </munder>
    <mfrac>
      <mi> a </mi>
      <mi> b </mi>
    </mfrac>
    <mover>
      <mo> ) </mo>
      <mo> &#x203E;<!--OVERLINE--> </mo>
    </mover>
  </mrow>
</math>

Also, not sure if this is an issue: The speech text output did not spell out the modifying characters: modified open paren with ¯ below, a over b, modified close paren with ‾ above.

Navigation not announcing "in"/"out of" in some cases

In a fraction, moving into the numerator/denominator should always say that it is in there. That's not happening all the time. In particularly, When moving at a char level, it doesn't announce it.

ClearSpeak Fraction options not being checked

Need to implement options for ClearSpeak fractions

"whereami" needs to take into account timing

Repeatedly issuing "whereami" within some small period of time should walk up the tree. That is not currently implemented.

This should be based on an estimate of how long it takes to speak the command along some delay (10 seconds?). The delay should probably be a user preference.

UEB capitalization needs work

Test case braille::UEB::iceb::hat_12_1_7 is: $A\hat{B}C$.

The output uses a cap word indicator but the example doesn't. This is likely because the last part of the definition in section GTM 12 says "the item is simply the previous individual symbol". Symbol in this case includes the cap indicator. So it seems in this case, $\hat{B}$ should not be considered a cap letter.

The code in braille.rs that builds a cap word from caps needs modification to handle "symbols".

Unexpected ClearSpeak and SimpleSpeak output for variables with both a subscript and superscript

MathCAT 0.1.23 when given the MathML for a variable with both a superscript and subscript generated unexpected ClearSpeak and SimpleSpeak output. For example,

Generated this ClearSpeak:
msub of, x comma k to the i-th power
and generated this SimpleSpeak:
msub of, x comma k to the i-th.
We expected something more like:
x sub k raised to the i-th power
Here is the source MathML (generated by MathJax for x^{i}_{k}):

<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
  <mstyle displaystyle="true" scriptlevel="0">
    <msubsup>
      <mi>x</mi>
      <mrow data-mjx-texclass="ORD">
        <mi>k</mi>
      </mrow>
      <mrow data-mjx-texclass="ORD">
        <mi>i</mi>
      </mrow>
    </msubsup>
  </mstyle>
</math>

The settings used were:

  set_preference("TTS".to_string(), "none".to_string()).unwrap();
  set_preference("Impairment".to_string(), "Blindness".to_string()).unwrap();
  set_preference("Language".to_string(), "en".to_string()).unwrap();
  set_preference("SpeechSound".to_string(), "None".to_string()).unwrap();
  set_preference("Verbosity".to_string(), "Verbose".to_string()).unwrap();
  set_preference("MathRate".to_string(), "100".to_string()).unwrap();
  set_preference("SpeechStyle".to_string(), "ClearSpeak".to_string()).unwrap();
  set_preference("SubjectArea".to_string(), "General".to_string()).unwrap();
  set_preference("Chemistry".to_string(), "SpellOut".to_string()).unwrap();
  set_preference("BrailleCode".to_string(), "Nemeth".to_string()).unwrap();

Absolute value not being read properly

In this example just above https://en.wikipedia.org/wiki/Fundamental_theorem_of_algebra#Complex-analytic_proofs, the absolute value is not being spoken for the first and last instances. Instead, it is saying "divides".

<math xmlns='http://www.w3.org/1998/Math/MathML' alttext='{\displaystyle {\tfrac {1}{2}}|z^{n}|<|p(z)|<{\tfrac {3}{2}}|z^{n}|}'>
<semantics>
  <mrow class='MJX-TeXAtom-ORD'>
    <mstyle scriptlevel='0' displaystyle='true'>
      <mrow class='MJX-TeXAtom-ORD'>
        <mstyle scriptlevel='0' displaystyle='false'>
          <mfrac>
            <mn>1</mn>
            <mn>2</mn>
          </mfrac>
        </mstyle>
      </mrow>
      <mrow class='MJX-TeXAtom-ORD'>
        <mo stretchy='false'>|</mo>
      </mrow>
      <msup>
        <mi>z</mi>
        <mrow class='MJX-TeXAtom-ORD'>
          <mi>n</mi>
        </mrow>
      </msup>
      <mrow class='MJX-TeXAtom-ORD'>
        <mo stretchy='false'>|</mo>
      </mrow>
      <mo>&lt;</mo>
      <mrow class='MJX-TeXAtom-ORD'>
        <mo stretchy='false'>|</mo>
      </mrow>
      <mi>p</mi>
      <mo stretchy='false'>(</mo>
      <mi>z</mi>
      <mo stretchy='false'>)</mo>
      <mrow class='MJX-TeXAtom-ORD'>
        <mo stretchy='false'>|</mo>
      </mrow>
      <mo>&lt;</mo>
      <mrow class='MJX-TeXAtom-ORD'>
        <mstyle scriptlevel='0' displaystyle='false'>
          <mfrac>
            <mn>3</mn>
            <mn>2</mn>
          </mfrac>
        </mstyle>
      </mrow>
      <mrow class='MJX-TeXAtom-ORD'>
        <mo stretchy='false'>|</mo>
      </mrow>
      <msup>
        <mi>z</mi>
        <mrow class='MJX-TeXAtom-ORD'>
          <mi>n</mi>
        </mrow>
      </msup>
      <mrow class='MJX-TeXAtom-ORD'>
        <mo stretchy='false'>|</mo>
      </mrow>
    </mstyle>
  </mrow>
  <annotation encoding='application/x-tex'>{\displaystyle {\tfrac {1}{2}}|z^{n}|&lt;|p(z)|&lt;{\tfrac {3}{2}}|z^{n}|}</annotation>
</semantics>
</math>

Handle rowspan/columnspan

Neither the speech rules nor the navigation rules work if rowspan or columnspan are given on an mtable element.

For navigation, it might be useful for the canonicalize step to mark row/col numbers and also ids of the up/down/before/after elements since those are hard to compute with xpath.

Navigation may also want to add a state variable for current row/col since moving to a spanned row/col potentially loses the current row/col when you move to a non-spanned cell.

Handle elementary math elements

MathCAT, lacks support for elementary math in:

• Speech
• Navigation
• Braille

The speech and navigation can probably be ported over from MathPlayer.

The braille requires a solution for displaying 2D braille.

OverView or Overview in preferences

In the Navigation section we have Overview and ResetOverView.

Consider whether the case usage should be consistent.

Typo in NSoiffer/MathCAT/blob/main/Rules/prefs.yaml

In the prefs.yaml file, it appears to have incorrect guidance in the comments

The ClearSpeak setting on line 17 is listed as follows:

      Roots: Auto               # PosNegSqRoot, RootEnd, PosNegSqRootEnd

However, "RootEnd" has no effect. Whereas "EndRoot" seems to.

Pattern match/replacement failure on aligned equations with empty columns?

Errors on aligned equations with empty columns

The three examples below generate the following error in MathCat Demo (using v0.1.17):

Pattern match/replacement failure!
caused by: attempting replacement pattern: "default" for "math".
Replacement
[test: [  if: 'x: "$MathRate = 100"' then:[x: "*"] else:[rate: 987654321.5[x: "*"]]]
]
...due to matching the following MathML with the pattern
x: "."

Example 1 - empty first column in row 2

\begin{equation}
\begin{split}
A & = \frac{\pi r^2}{2} \\
 & = \frac{1}{2} \pi r^2
\end{split}
\end{equation}

Example 2 - empty first column in rows 2, 3

\begin{align}
  \frac{d}{dx} \ln x &= \lim_{h\to 0} \frac{\ln(x+h) - \ln x}{h} \\
   &= \ln e^{1/x}\\
   &= \frac{1}{x}
 \end{align}

Example 3 - empty first column in row 2

\begin{aligned}
    x &=\frac{a}{c} + \frac{b}{c} \\
     &=\frac{a+b}{c} 
\end{aligned}

Example 4 - empty third column in rows 2, 3

\begin{align}
  \frac{d}{dx} \ln x &= \lim_{h\to 0} \frac{\ln(x+h) - \ln x}{h} \\
 \frac{d}{dx} \ln x  &= \ln e^{1/x} && \text{How this follows is left as an exercise.}\\
 \frac{d}{dx} \ln x  &= \frac{1}{x} && \text{Using the definition of ln as inverse function}
 \end{align}

If the columns each have content, the error goes away:

Example 1 works with right hand repeated in col 1

\begin{equation}
\begin{split}
A & = \frac{\pi r^2}{2} \\
A & = \frac{1}{2} \pi r^2
\end{split}
\end{equation}

Example 2 works with right hand repeated in col 1

\begin{align}
  \frac{d}{dx} \ln x &= \lim_{h\to 0} \frac{\ln(x+h) - \ln x}{h} \\
   \frac{d}{dx} \ln x &= \ln e^{1/x}\\
   \frac{d}{dx} \ln x &= \frac{1}{x}
 \end{align}

Example 3 works with right hand repeated in col 1

\begin{aligned}
    x &=\frac{a}{c} + \frac{b}{c} \\
    x &=\frac{a+b}{c} 
\end{aligned}

Example 4 works when adding dummy text content to col 3 in rows 2, 3

\begin{align}
  \frac{d}{dx} \ln x &= \lim_{h\to 0} \frac{\ln(x+h) - \ln x}{h} \\
 \frac{d}{dx} \ln x  &= \ln e^{1/x} & \text{dummy} & \text{How this follows is left as an exercise.}\\
 \frac{d}{dx} \ln x  &= \frac{1}{x} & \text{dummy} & \text{Using the definition of ln as inverse function}
 \end{align}

SpeechOverrides_CapitalLetters and "spell" command

For chemistry, the rule for chemical elements is:

    - spell: text()

That works well most of the time. However, if the user pref SpeechOverrides_CapitalLetters is set to a string such as cap, then the word cap gets spelled out also. When the tts is none, that results in c a p and when it is set to a value such ass ssml, you get for S

<say-as interpret-as='characters'>cap <say-as interpret-as='characters'>s</say-as></say-as>

due to the unicode.yaml rule

 - "B-Z": 
    - test:
        # note: processing of ranges converts '.' into the character, so it needs to be in quotes below
        if: "$SpeechOverrides_CapitalLetters = ''"
        then: [spell: "'.'"]
        else: [x: "$SpeechOverrides_CapitalLetters", spell: "translate('.', 'BCDEFGHIJKLMNOPQRSTUVWXYZ', 'bcdefghijklmnopqrstuvwxyz')"]

Because the special rule for 'a' in English, you can't move implementation of $SpeechOverrides_CapitalLetters into the implementation of spell. Maybe add a check for "English" in the spell implementation?

An alternative might be to add a no_spell rule to turn off spelling.

Missing speech text for "top brace" and "bottom brace" symbols

The ClearSpeak output for math with a top or bottom brace includes the ASCII character for the symbol instead of spelling out the name of the symbol. For example, for:

MathCat 0.1.23 generated the following ClearSpeak:

modified x plus y plus z with ⏞ above

We expected something like:

modified x plus y plus z with top brace above

The MathML was:

<math xmlns="http://www.w3.org/1998/Math/MathML">
    <mover>
      <mrow>
        <mi>x</mi>
        <mo>+</mo>
        <mi>y</mi>
        <mo>+</mo>
        <mi>z</mi>
      </mrow>
      <mo>&#x23DE;</mo>
    </mover>
</math>

The bottom brace example:

generated the following ClearSpeak:

modified x plus y plus z with ⏟ below

We expected something like:

modified x plus y plus z with bottom brace below

The MathML was:

<math xmlns="http://www.w3.org/1998/Math/MathML">
    <munder>
      <mrow>
        <mi>x</mi>
        <mo>+</mo>
        <mi>y</mi>
        <mo>+</mo>
        <mi>z</mi>
      </mrow>
      <mo>&#x23DF;</mo>
    </munder>
</math>

UEB bug while navigating

Start 4ac. Zoom in the '4'. Then move right to the a. The G1 indicator (⠰) disappears. If you move right to the c, it reappears.

MathCAT crashes on some MathML expressions.

To reproduce:

Using the NVDA MathCAT add-on, attempt to read a MathML expression that causes a crash.

NVDA emits an error tone, and leaves a Python traceback in the log.

Perhaps there's a separate issue for the add-on: would it be possible to log the error without causing a Python traceback (e.g., catch the error, log it and continue)? A major drawback of the current implementation is that the user can't even navigate past the problematic MathML expression to continue reading the document (e.g., with the Down-Arrow key).

The first example is taken from the Wikipedia article on the Fundamental Theorem of Calculus:
https://en.wikipedia.org/wiki/Fundamental_theorem_of_calculus

The second example appears in the Wikipedia entry on elementary algebra:
https://en.wikipedia.org/wiki/Elementary_algebra

Zip file containing instances of the MathML that crash and also the corresponding log files:
mathcat-crashes.zip