Use Unicode in Typst

[gzagatti]

Use Unicode in Typst

[jgm]

No, this can't be done with a filter. A filter can only play with the pandoc AST; it can't adjust the output of texmath functions.

[jgm]

No, there isn't. In principle, it could be added, but this would require:

• an extra parameter on the texmath typst reader to determine whether symbol names should be used
• a new pandoc extension that toggles this

On balance I don't think it's worth the effort and extra complexity.

[gzagatti]

Thanks for your response!

I would like to expand on my initial request to provide more context.

Since math rendering in HTML supports so many flags, I thought it would be trivial to include a --unicode flag for math that would also be available for Typst.

I do find it a bit odd that even ASCII characters like +, - and = get converted to Typst symbol names. Writing equations with words is like writing Chinese with pinying. It is possible but not expressive. You might appreciate the Chinese poem "Lion-Eating Poet" to know what I mean.

My understanding is that Typst named symbols are just shortcuts to the underlying Unicode characters. I had the chance to look at the texmath source code. Indeed, it seems that under the hood all math symbols get first converted to Unicode and then to Typst symbol names. I changed the offending function from

-- src/Text/TeXMath/Writers/Typst.hs#L58
writeExp (ESymbol _t s) =
  maybe (esc s) id $ M.lookup s typstSymbolMap

to

writeExp (ESymbol _t s) = esc s

Then, I recompiled pandoc with the patched dependency to obtain the desired effect. It works pretty well except for \setdiff which maps to \ and should be escaped to \\. Given that I have no experience with Haskell, it took me much longer to figure out how to recompile pandoc with the patched dependency then to patch the function itself. Also thanks to the well-organized and documented source code, it was easy to locate the issue.

If my understanding that Typst uses Unicode is correct, then converting symbols to Unicode is much less complex and error prone. Typst symbols are more likely to change than the Unicode characters, so Unicode would be more robust.

However, if changes in this direction add unnecessary complexity and, especially, requires too much effort I agree that it's not worth the investment. Although, it would be a missed opportunity.

[jgm]

I do find it a bit odd that even ASCII characters like +, - and = get converted to Typst symbol names.

Yes, that's definitely odd, and I can change that in the default behavior. (Avoiding symbol names when the code point is in the ASCII range.)

The change is certainly possible, but it's not trivial. We'd need some mechanism in pandoc to communicate the option to the typst writer in texmath. I would prefer to avoid a new command line option. And I would prefer to avoid any new complexity; that's where bugs come from. Producing typst that resembles examples from the typst documentation seems like a good goal, and that's what we aim at.

[gzagatti]

I agree that Typst documentation tends to favour the use of named symbols, even though it natively supports UTF-8 characters and named symbols are nothing more than sugar for their UTF-8 representation. I believe that this is to cater to users who might find it difficult to input non-ASCII characters, are coming from LaTex and/or might have problems installing UTF-8 fonts.

The only mention to UTF-8 input in Typst documentation is in this section of the documentation which is one the things that sold Typst to me.

You can use all Typst functions within math mode and insert any content. If you want them to work normally, with code mode in the argument list, you can prefix their call with a #. Nobody can stop you from using rectangles or emoji as your variables anymore:

$ sum^10_(🥸=1)
 #rect(width: 4mm, height: 2mm)/🥸
 = 🧠 maltese $

If you'd like to enter your mathematical symbols directly as Unicode, that is possible, too!

In any case, Typst documentation is probably outside of the scope of this issue. If you have to make a choice, I agree that you have to pick one option consistently.

However, is there anyway I could write a Pandoc filter to avoid calling the conversion to named symbols in the function below?

-- src/Text/TeXMath/Writers/Typst.hs#L58
writeExp (ESymbol _t s) =
  maybe (esc s) id $ M.lookup s typstSymbolMap

I would be happy to live with a Pandoc filter. Certainly better than a patched Pandoc. I'm just not sure how to go about it. Any advice would be super-helpful.