gfngfn/satysfi

A statically-typed, functional typesetting system

日本語版 README はこちら

Summary of SATySFi

SATySFi (pronounced in the same way as the verb “satisfy” in English) is a new typesetting system equipped with a statically-typed, functional programming language. It consists mainly of two “layers” ― the text layer and the program layer. The former is for writing documents in LaTeX-like syntax. The latter, which has OCaml-like syntax, is for defining functions and commands. SATySFi enables you to write documents markuped with flexible commands of your own making. In addition, its informative type error reporting will be a good help to your writing.

This software was supported by:

• IPA Mitou Project 2017 (June 2017 – February 2018; see the abstract here written in Japanese),
• Dwango Co., Ltd. (October 2018 – March 2019; as a part-time job), and
• many anonymous supporters who bought The SATySFi​book,

and its development continues to this day (Feb 2024).

Install using Satyrographos (for non-devs)

You can install SATySFi with package manager Satyrographos.

# For Ubuntu 20.04
sudo apt-get update
sudo apt-get install build-essential git m4 unzip curl pkg-config
sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)

# For Mac
# Please make sure homebrew is installed. Otherwise, follow https://brew.sh/
brew update
brew install opam

# Common: Set up OPAM
opam init
eval $(opam env)
opam repository add satysfi-external https://github.com/gfngfn/satysfi-external-repo.git
opam repository add satyrographos https://github.com/na4zagin3/satyrographos-repo.git
opam update

# Common: Install SATySFi
opam depext satysfi satysfi-dist satyrographos
opam install satysfi satysfi-dist satyrographos

# Common: Set up the SATySFi standard library
satyrographos install

Install using OPAM

Prerequisites

Here is a list of minimally required softwares.

• bzip2
• cc
• git
• m4
• make
• unzip
• wget or curl
• opam 2
  • See https://opam.ocaml.org/doc/Install.html.
  • On Windows, Windows Subsystem for Linux (WSL) 2 is recommended to avoid bubblewrap-related issues.
• ocaml 4.10.0 (installed by OPAM)

Also, we must add an external OPAM repo to build. This can be done by the following command.

opam repository add satysfi-external https://github.com/gfngfn/satysfi-external-repo.git
opam update

Example (Ubuntu)

sudo apt-get update
sudo apt-get install build-essential git m4 unzip curl

sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)

# The following command will ask if you allow OPAM to modify some files (e.g. ~/.bash_profile).
# Be sure to read its instructions. Otherwise, some environment variables won't be set.
opam init --comp 4.10.0

eval $(opam env)

opam repository add satysfi-external https://github.com/gfngfn/satysfi-external-repo.git
opam update

Example (OS X Mavericks or later)

# Before running this scripts, install essential softwares such as GCC and Make. They can be installed from Xcode Command Line Tools.
# Also, install Homebrew.

brew update
brew install opam

# The following command will ask if OPAM modifies some files.
# Be sure to read their instructions. Otherwise, some environment variables won't be set.
opam init --comp 4.10.0

eval $(opam env)

opam repository add satysfi-external https://github.com/gfngfn/satysfi-external-repo.git
opam update

Build

First, clone this repository and submodules. Then build SATySFi using OPAM.

# clone
git clone https://github.com/gfngfn/SATySFi.git
cd SATySFi

# build
opam pin add satysfi .
opam install satysfi

• To reinstall, run opam reinstall satysfi.
• To uninstall, run opam uninstall satysfi.

Setup for SATySFi

Before using SATySFi, one should put libraries and fonts onto the appropriate directory. This can be done by invoking the following commands in order:

./download-fonts.sh
./install-libs.sh

The former downloads the fonts required by the default settings into lib-satysfi/dist/fonts/, and the latter copies lib-satysfi/ to /usr/local/share/satysfi/.

During this setup, the following fonts are downloaded. Consult their license before using them.

• Junicode
• IPA Font
• Latin Modern
• Latin Modern Math

Usage of SATySFi

Type

satysfi <input file> -o <output file>

in order to convert <input file> into <output file>. For example, when you want to convert doc.saty into output.pdf, the following command will work:

satysfi doc.saty -o output.pdf

Starting out

First of all, let’s try to compile the demo file. It is in demo folder. Because this demo file has MakeFile, All you should do is only type make.

cd demo
make

If demo.pdf is created, then the setup has been finished correctly.

Reference

In addition, a concice reference of SATySFi is written by SATySFi itself in doc folder. You need to compile it to read.

cd doc
make

Command-line options

• -v, --version: Prints the version.
• -o, --output: Specify the name of the output PDF file. if this option is not given explicitly, the name of the output file is the concatenation of the base name of the input file and the extension .pdf.
• -b, --bytecomp: Use byte compiler and enhance performance of computation.
• --full-path: Displays file names with their absolute path when outputting them to stdout.
• --type-check-only: Stops after type checking.
• --debug-show-bbox: Outputs bounding boxes for each glyph (for the purpose of debugging).
• --debug-show-space: Outputs boxes for spaces (for the purpose of debugging).

Learning SATySFi

Wiki (currently written only in Japanese) has some information about learning SATySFi.

satysfi's Issues

Fix post-install message

Problem & Proposal

The post-install message introduced by ab3f781 is wrong. $(PREFIX_LIB) must be replaced by a real path.

Please set SATYSFI_LIB_ROOT. You may want to add the following line to ~/.bashrc"
export SATYSFI_LIB_ROOT=$(PREFIX_LIB)/lib-satysfi"

Environment

macOS 10.13.2
OCaml 4.05.0 + OPAM 1.2.2
SATySFi commit bc30f33

Workaround

• Nothing.

Should we manually install Arno Pro fonts?

Problem

After installing SATySFi and setting $SATYSFI_LIB_ROOT, I want to build the document in /doc. However, running make command is failed due to the lack of a font file. The following is the error message.

! [Error] cannot load font file '/home/nek/.opam/4.05.0/lib-satysfi/dist/fonts/ArnoPro-Regular.otf';
      /home/nek/.opam/4.05.0/lib-satysfi/dist/fonts/ArnoPro-Regular.otf: No such file or directory

Should we manually install this font? Or, this font is not supported yet?

How to reproduce

$ opam install satysfi
$ export SATYSFI_LIB_ROOT=/home/nek/.opam/4.05.0/lib-satysfi  # set your library path
$ cd doc
$ make

Full output of make is here.

Environment

• Ubuntu 17.10 (Artful Aardvark)
• OCaml 4.05.0 + OPAM 1.2.2
• SATySFi commit bc30f33 + glibc-2.26 patch (#11)

Travis started to fail

Currently #44 (build #929) and #45 (build #930) fail (for seemingly the same reasons). As #44 is a trivial change, they must be spurious errors.

I couldn't figure out the exact cause. Seems to be related to core_kernel.

How about using labels to categorize issues?

It seems that there are some different types of issues, such as [bug], [enhancement] and [language-design].
Categorizing issues will help you when the number of issues increase. The repository of TypeScript is a good example.

If you don't feel necessity of this proposal, feel free to close this.

Label: [meta]

Sandboxing of OPAM 2 prevents the build from downloading font files

OPAM2 now has sandboxing: the build cannot access the external resources. SATySFi build fails at the downloading of font files.

See ocaml/opam#3460 for a discussion of the sandboxing.

In the following build attempt, curl fails due to the name resolution failure of www.gust.org.pl.

$ opam --version
2.0.0
$ opam switch show
/Users/jun/.share/4.06.1-satysfi
$ opam pin add .
[NOTE] Package satysfi is currently pinned to
       git+file:///Users/jun/.share/4.06.1-satysfi/SATySFi#master (version
       0.0.2).
[satysfi.0.0.2: git]
[satysfi.0.0.2: git]
[satysfi.0.0.2: git]
[satysfi.0.0.2: git]
[satysfi.0.0.2: git]
[satysfi.0.0.2: git]
[satysfi.0.0.2] synchronised from git+file:///Users/jun/.share/4.06.1-satysfi/a/SATySFi#master
satysfi is now pinned to git+file:///Users/jun/.share/4.06.1-satysfi/a/SATySFi#master (version 0.0.2)
The following actions will be performed:
  ∗ install satysfi 0.0.2*
Do you want to continue? [Y/n] 
y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫 
[ERROR] The compilation of satysfi failed at
        "/Users/jun/.opam/opam-init/hooks/sandbox.sh build make -f Makefile lib
        PREFIX=/Users/jun/.share/4.06.1-satysfi/_opam".

#=== ERROR while compiling satysfi.0.0.2 ======================================#
# context              2.0.0 | macos/x86_64 | ocaml-base-compiler.4.06.1 | pinned(git+file:///Users/jun/.share/4.06.1-satysfi/a/SATySFi#master#285ff60f)
# path                 ~/.share/4.06.1-satysfi/_opam/.opam-switch/build/satysfi.0.0.2
# command              ~/.opam/opam-init/hooks/sandbox.sh build make -f Makefile lib PREFIX=/Users/jun/.share/4.06.1-satysfi/_opam
# exit-code            2
# env-file             ~/.opam/log/satysfi-87558-335865.env
# output-file          ~/.opam/log/satysfi-87558-335865.out
### output ###
# if [ -x "$(command -v curl)" ]; then \
# [...]
# 	else \
# 	  wget -N http://www.gust.org.pl/projects/e-foundry/latin-modern/download/lm2.004otf.zip -P temp/; \
# 	  wget -N http://www.gust.org.pl/projects/e-foundry/lm-math/download/latinmodern-math-1959.zip -P temp/; \
#         fi
#   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0curl: (6) Could not resolve host: www.gust.org.pl
#   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0curl: (6) Could not resolve host: www.gust.org.pl
# make: *** [lib] Error 6



<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫 
┌─ The following actions failed
│ λ build satysfi 0.0.2
└─ 
╶─ No changes have been performed

Switch to dune

Support OCaml >= 4.06.0

consider using ocamlgraph for some graph structure implementation

seems ocamlgraph's directed graph can be used to replace src/backend/flowGraph.ml

Consider adding a polymorphic version of `itemize`

This issue originates from a comment by @hak7a3 on the type itemize of SATySFi:

#SATySFi 、itemizeが組み込み型なんだけど、各Itemにオプション引数ないから辛い（bibitemっぽいことやろうとした）。
現状、新しい文法レベルの要素を追加するには、srcいじるか、型追加するしかない？

— HAKUTA Shizuya (@hak7a3) 2018年2月13日

It basically says that the type itemize of SATySFi seems not so suitable for making a slightly elaborate itemization like the thebibliography environment of LaTeX.

Adding a new type poly-itemize defined by:

type 'a poly-itemize = PolyItem of 'a * inline-text * ('a poly-itemize) list

and a syntax sugar for it probably enhances the user-friendliness of the language.

(Unix.Unix_error "Illegal seek" sysconf "") with glibc-2.26

How to reproduce

With glibc-2.26,

$ git clone --recursive https://github.com/gfngfn/SATySFi.git
$ cd SATySFi
$ opam pin add satysfi .
$ opam install satysfi
$ cd doc
$ make

Then, an error (Unix.Unix_error "Illegal seek" sysconf "") occurred:

$ make
satysfi doc-primitives.saty -o primitives.pdf
Uncaught exception:
  
  (Unix.Unix_error "Illegal seek" sysconf "")

Raised by primitive operation at file "src/core_unix.ml", line 348, characters 14-29
Makefile:4: recipe for target 'all' failed
make: *** [all] Error 2

Environment

• Ubuntu 17.10 (Artful Aardvark)
• Ubuntu GLIBC 2.26-0ubuntu2.1
• OCaml 4.05.0 (opam 1.2.2)
• SATySFi commit 3fb9591

How to fix

This bug is reported to janestreet/core#95. Updating core to v0.10.0 fixes the problem (currently, opam config file pins its version to v0.9.1).

Broken embedded text information

The current version of SATySFi doesn't correctly produce embedded texts (i.e. information used when copying or searching on PDF viewers). This is a known issue, which is caused by lack of the /ToUnicode entry.

Fuzzing with American fuzzy lop (afl)

The OCaml compiler supports afl instrumentation (see Chapter 23 Fuzzing with afl-fuzz for details) and it may benefit SATySFi for finding crash bugs.

Feature request: module open

I want to introduce a new primitive to open a module.

For example, you can open a module in OCaml:

module M = struct let f () = 42 end
open M
(* We can use `f` here, instead of `M.f` *)

Unlike OCaml, SATySFi does not create a module per a file (at least now). So the necessity of open is less for SATySFi, but we sometimes need this (e.g., when using a signature to hide implementation).

Missing dependency of uchar

I noticed that a package uchar is not written in opam whereas it's needed for the external package otfm (ref). Also, a constraint "uutf" {>= "1.0.0"} is missing.

Although uchar is unnecessary for OCaml >= 4.03, this should be written in opam.
Or, using opam repo (#14) will help.

Cannot deal with different variant types that have the same name

Since the current implementation of Macrodown distinguishes variant types only by names, it CANNOT deal correctly with programs like:

type hoge = A | B

let f x = match x with A -> 0 | B -> 1

type hoge = X | Y

let y = f X in
  arabic y

This program should be rejected as an ill-typed one, but is indeed allowed by the current implementation of the typechecker.

Inconsistency about the `/ColorSpace` entry for embedded JPEG images

https://twitter.com/kurgm/status/963657925402165249

let-{inline,block} in arguments

In function arguments, we can use let, let-rec, and let-math. However, we cannot use let-inline or let-block (syntax error at parser). Is this a design choice of SATySFi, or a bug?

Example

This is allowed:

+math (
  let-math \foo = ${foo} in
  ${ \foo }
);

but this is disallowed:

+p (
  let-inline \bar = {bar} in  % Syntax Error in this line
  { \bar; }
);

Environment

• Ubuntu 18.04
• OCaml 4.06.1
• SATySFi >=0.03 (current head ccbe682)

Odd behaviour of type with multiple parameters

It seems that the type-inference for multi-parameter types treats its argument as if it's reversed.

A minimal example:

module Multiarg : sig
  type 'k 'v pair

  val singleton : 'k -> 'v -> 'k 'v pair
end = struct
  type 'k 'v pair = Pair of 'k * 'v

  let singleton k v = Pair (k, v)
end

! [Type Error] at line 1, character 0 to line 9, character 3:
    The implementation of value 'singleton' has type
      '#a -> '#b -> '#b '#a Multiarg.pair
    which is inconsistent with the type required by the signature
      '#a -> '#b -> '#a '#b Multiarg.pair

The most recent SATySFi fails to build on macOS

The most recent SATySFi in master (871c286) fails to build on macOS with the following errors:

=-=- Synchronising pinned packages =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫 
[satysfi] /Users/me/github/SATySFi/ synchronized
The following actions will be performed:
  ∗  install satysfi *

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫 
[satysfi.~unknown] /Users/me/github/SATySFi/ already up-to-date

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫 
[ERROR] The compilation of satysfi failed at "make -f Makefile
        PREFIX=/Users/me/.opam/4.06.0".
Processing  1/1: [satysfi: make Makefile]
#=== ERROR while installing satysfi.~unknown ==================================#
# opam-version 1.2.2
# os           darwin
# command      make -f Makefile PREFIX=/Users/me/.opam/4.06.0
# path         /Users/me/.opam/4.06.0/build/satysfi.~unknown
# compiler     4.06.0
# exit-code    2
# env-file     /Users/me/.opam/4.06.0/build/satysfi.~unknown/satysfi-41096-05296d.env
# stdout-file  /Users/me/.opam/4.06.0/build/satysfi.~unknown/satysfi-41096-05296d.out
# stderr-file  /Users/me/.opam/4.06.0/build/satysfi.~unknown/satysfi-41096-05296d.err
### stdout ###
# Hint: Recursive traversal of subdirectories was not enabled for this build,
# [...]
#   '_tags' or 'myocamlbuild.ml' file). If you have modules in subdirectories,
#   you should add the option "-r" or create an empty '_tags' file.
#   
#   To enable recursive traversal for some subdirectories only, you can use the
#   following '_tags' file:
#   
#       true: -traverse
#       <dir1> or <dir2>: traverse
#       
### stderr ###
# make: *** [all] Error 10



=-=- Error report -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫 
The following actions failed
  ∗  install satysfi ~unknown

Here is the log files generated by the opam.

Environment

macOS High Sierra 10.13.4

$ opam switch
--     -- 3.07    Official 3.07 release
... [snip] ...
4.06.0  C 4.06.0  Official 4.06.0 release
--     -- 4.06.1  Official 4.06.1 release
system  I system  System compiler (4.06.0)
# 197 more patched or experimental compilers, use '--all' to show

$ opam --version
1.2.2

$ opam repository
  10 [git]   satysfi-external     https://github.com/gfngfn/satysfi-external-repo.git
   0 [http]     default     https://opam.ocaml.org

Enhancement: Optional arguments of let-{rec,inline,block}

I noticed that optional arguments are sometimes not supported (syntax error).

Example:

% These are good.
let f ?:x y = ()
let-inline ctx \f ?:x y = read-inline ctx {}
let-block ctx +f ?:x y = read-block ctx '<>

% These are not.
let-rec f ?:x y = ()
let-inline \f ?:x y = {}
let-block +f ?:x y = '<>

So I asked this on Twitter in Japanese, and gfn-san answered that these three should be implemented. Therefore I post this issue for a record.

Environment

• Ubuntu 18.04
• OCaml 4.06.1
• SATySFi >=0.03 (current head 7ba1d05)

Can we shadow primitive types?

I noticed that the following code produces an odd error.

type int = A
module M : sig
  val a : A
end = struct
  let a = A
end

$ satysfi --type-check-only test.saty
 ---- ---- ---- ----
  target file: 'test.pdf'
  dump file: 'test.satysfi-aux' (will be created)
  parsing 'test.saty' ...
 ---- ---- ---- ----
  reading 'test.saty' ...
! [Type Error] at line 3, character 0 to line 7, character 3:
    The implementation of value 'a' has type
      int
    which is inconsistent with the type required by the signature
      int

This behavior may be related to the specification of shadowing of types. SATySFi can shadow user-defined types as follows. But can we shadow primitive types?

type t = A
module M : sig
  val a : t
end = struct
  let a = A
end

type t = B
module N : sig
  val a : t
end = struct
  let a = B
  % let a = A  % This produces a type error
end

Note that OCaml can shadow primitive types.

Pushing back patches of external dependencies

Is there a plan to push patches in gfngfn/camlpdf and gfngfn/otfm back to their upstreams, johnwhitington/camlpdf and dbuenzli/otfm? I can help preparing PRs.

Change show-* options of stdjabook to be optional

Both stdja.satyh and stdjabook.satyh have show-toc option and show-title option.

Currently they cannot be omitted because there're no default values of them. So how about changing them to be optional like PR #111?

If gfn-san doesn't prefer such a change, feel free to close this issue :)

Type constructor order

As far as I know, there are three common syntaxes for type constructors and one:

• map(k, v): C++, Reason, Rust etc.
• map k v: Haskell, OCaml (revised syntax), Coq etc.
• (k, v) map: OCaml (normal syntax), SML, etc.
• k v map: SATySFi

I suggest changing the syntax to Haskell-like one, which is uniform to SATySFi's expressions.

Is this a bug of a parser: `module M struct = ... end **in** document`?

Currently, it is necessary to write in before a document-type value in a .saty file even if there is only a definition of a module in "preamble".

@require: stdjareport

module M = struct
  let foo = 42
end

in  % <-- This `in` is necessary! Without this, SATySFi throws a syntax error.
document (|
  title = {example};
  author = {nekketsuuu};
|) '<
>

It looks strange a bit if I think like OCaml. Is this a design choice of SATySFi, or a bug of parser?

Environment

• Ubuntu 18.04
• OCaml 4.06.1
• SATySFi >=0.03 (current head 7ba1d05)

Nested partial application of polymorphic functions results in Stack Overflow at compile time

If we nest the partial applications of a polymorphic function and expose at least two of them, then compilation fails with stack overflow.

For example, consider the following.

crasher.satyh:

module Crasher : sig
  type 'a thing
  % val pp-thing-dummy  : unit -> ('a -> string) -> 'a thing -> string
  val pp-thing-with     :         ('a -> string) -> 'a thing -> string
  val pp-thing          :                           'a thing -> string
end = struct
  type 'a thing = Wrap of 'a

  let pp-thing-dummy d pp tb =
    match tb with
    | Wrap(b) -> pp b

  let pp-thing-with = pp-thing-dummy ()
  let pp-thing = pp-thing-with (fun a -> `_`)
end

crash.saty:

@require: stdja
@import: crasher

StdJa.document
  (| title = {}; author = {}
   ; show-title = false
   ; show-toc = false |)
   '<
 +p { finished. }
>

Then, compiler halts with the following error:

 ---- ---- ---- ----
  target file: 'crash.pdf'
  dump file: 'crash.satysfi-aux' (already exists)
  parsing 'crash.saty' ...
  parsing 'stdja.satyh' ...
  parsing 'pervasives.satyh' ...
  parsing 'gr.satyh' ...
  parsing 'list.satyh' ...
  parsing 'math.satyh' ...
  parsing 'color.satyh' ...
  parsing 'crasher.satyh' ...
 ---- ---- ---- ----
  reading 'pervasives.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'list.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'crasher.satyh' ...
Uncaught exception:
  
  Stack overflow

Raised at file "queue.ml", line 68, characters 11-22
Called from file "src/frontend/directedGraph.ml", line 175, characters 22-34

I think this case is near-minimal, because:

• If we replace 'a in signatures with monomorphic type, say unit, the compile error disappears:

  val pp-thing-with     : (unit -> string) -> unit thing -> string
  val pp-thing          :                   unit thing -> string

• If we comment out at least one of val pp-thing: ... and val: pp-thing-with: ..., compilation success.
  Furthermore, if we comment out val pp-thing: ... and uncomment val: pp-thing-dummy, compilation also succeeds.

Fails to compile x'_1^2

Preparing release notes for SATySFi for windows 20180707 I found that not all of the following compile due to missing clauses for mathtop.

  +math-list[${ x^2 }];
  +math-list[${ x_1 }];
  +math-list[${ x^2_1 }];
  +math-list[${ x_2^1 }];
  +math-list[${ x'^2 }];
  +math-list[${ x'_1 }];
  %+math-list[${ x'^2_1 }];  % error
  %+math-list[${ x'_2^1 }];  % error

Switch to comma-separated lists?

Semicolon-separated lists [a; b; c] are a common pitfall in OCaml. As SATySFi requires parentheses on tuples (a, b), don't we switch to comma-separated lists [a, b, c]?

Pros:

• Consistent with common languages: C (initializers), JavaScript, Python, Haskell, SML, Reason, etc.
• The symbol ; will be free

Possible cons I came up so far:

• OCaml (both normal and revised syntax) and Coq at least employ semicolon-separated lists. Note that SATySFi's program part largely resembles OCaml.
• Backward compatibility with existing SATySFi codes.

How about using annotated tags for releases?

Currently, git tags v0.0.1, v0.0.2, and v0.0.3 are all lightweight tags. But it's better to use annotated tags for releases.

@gfngfn Would you use annotated tags for future releases?

• Ref.
  • How can I tell if a given git tag is annotated or lightweight? -- Stack Overflow
  • In what circumstances should I add the -a flag to the git tag command? -- Stack Overflow
  • How to annotate an existing tag in git -- Stack Overflow
  • dune subst and git lightweight tags -- OCaml Discuss

VM causes strange stack overflow (while naïve interpreter does NOT)

Typesetting the following document foo.saty by invoking the command$ satysfi --bytecomp foo.saty fails due to stack overflow, while typesetting it by $ satysfi foo.saty does NOT:

@require: stdja

let-inline \repeat n it =
  let-rec aux n it =
    if n <= 0 then {} else
      let it-after = aux (n - 1) it in {#it;#it-after;}
  in
    aux n it
in

document (|
  title = {Sample Document}; author = {gfn};
  show-title = false; show-toc = false;
|) '<
  +p{\repeat(2){foo}}
>

The stdout and stderr is as follows:

 ---- ---- ---- ----
  target file: 'foo.pdf'
  dump file: 'foo.satysfi-aux' (will be created)
  parsing 'foo.saty' ...
  parsing 'stdja.satyh' ...
  parsing 'pervasives.satyh' ...
  parsing 'gr.satyh' ...
  parsing 'geom.satyh' ...
  parsing 'list.satyh' ...
  parsing 'math.satyh' ...
  parsing 'color.satyh' ...
 ---- ---- ---- ----
  reading 'pervasives.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'list.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'color.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'geom.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'math.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'gr.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'stdja.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'foo.saty' ...
  type check passed. (document)
 ---- ---- ---- ----
  evaluating texts ...
Uncaught exception:

  Stack overflow

Raised at file "lib/read.mll", line 712, characters 13-36
Called from file "lib/read.mll", line 241, characters 23-47

A new syntax sugar for math list

I consider introducing a new syntax of the form

${|foo|bar|baz|}

as an alias for

[${foo}; ${bar}; ${baz}]

Switch from uCoreLib to uucp and utext

Daniel just released his another Unicode library: https://github.com/dbuenzli/utext

As a side note, I recommend using packages used by Mirage organization or packages developed by people from OCaml Labs since those packages are well documented and well maintained

0-ary parallel definition is not accepted

let x = y and y = {a} in x
    %=> {a}

let f = g and g x = x + 1 in f 2
    %=> 3

Though previous version accepted these kinds of code, current version does not (Assert_failure exception occurs during the evaluation because of the absence of a variable in the environment). Probably I broke this system while attempting to adopt it to the module system.

How to make table

How to write '.mcrdh' code to make table with following data
[
['foo','bar','baz'],
['fuga','hoge','<'],
['^', 'fizz', 'buzz']
]

to HTML as

foo	bar	baz
fuga	hoge
	fizz	buzz

?

Probably, is '.mcrd' like followings?
\tbody{|
| \tr{|foo|bar|baz|}
| \tr{|fuga|hoge|<|}
| \tr{|^|fizz|buzz|}
|}

Please advise. Thank you.

autoconf is required to install SATySFi, autoconf is not preinstalled on macOS and Ubuntu

Hello I love SATySFi so much. If we solve this problem, SATySFi must be much better!

Problem

I tried to install SATySFi on macOS High Sierra(10.13.2), however opam install satysfi failed with below errors.

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
[ERROR] The compilation of conf-autoconf failed at "which autoconf".

#=== ERROR while installing conf-autoconf.0.1 =================================#
# opam-version 1.2.2
# os           darwin
# command      which autoconf
# path         /Users/syrup/.opam/4.06.0/build/conf-autoconf.0.1
# compiler     4.06.0
# exit-code    1
# env-file     /Users/syrup/.opam/4.06.0/build/conf-autoconf.0.1/conf-autoconf-18608-ffb3fd.env
# stdout-file  /Users/syrup/.opam/4.06.0/build/conf-autoconf.0.1/conf-autoconf-18608-ffb3fd.out
# stderr-file  /Users/syrup/.opam/4.06.0/build/conf-autoconf.0.1/conf-autoconf-18608-ffb3fd.err



=-=- Error report -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
The following actions were aborted
  ∗  install bitv    1.3
  ∗  install satysfi ~unknown
The following actions failed
  ∗  install conf-autoconf 0.1
No changes have been performed

=-=- conf-autoconf.0.1 troobleshooting =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
=> This package relies on external (system) dependencies that may be missing. `opam depext
   conf-autoconf.0.1' may help you find the correct installation for your system.

Solution

Just execute brew install autoconf

Proposal

Add brew install autoconf in README.md

Byte-compiler supported version won't compile on macOS with ruby-2.5.0

Following the following code on macOS High Sierra 10.13.4, it fails to compile the SATySFi:

$ opam update
$ opam install SATySFi

=-=- Synchronising pinned packages =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫 
[satysfi] /Users/me/github/SATySFi/ already up-to-date
The following actions will be performed:
  ∗  install satysfi *

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫 
[satysfi.~unknown] /Users/me/github/SATySFi/ already up-to-date

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫 
[ERROR] The compilation of satysfi failed at "make -f Makefile
        PREFIX=/Users/me/.opam/4.06.0".
Processing  1/1: [satysfi: make Makefile]
#=== ERROR while installing satysfi.~unknown ==================================#
# opam-version 1.2.2
# os           darwin
# command      make -f Makefile PREFIX=/Users/me/.opam/4.06.0
# path         /Users/me/.opam/4.06.0/build/satysfi.~unknown
# compiler     4.06.0
# exit-code    2
# env-file     /Users/me/.opam/4.06.0/build/satysfi.~unknown/satysfi-38561-05296d.env
# stdout-file  /Users/me/.opam/4.06.0/build/satysfi.~unknown/satysfi-38561-05296d.out
# stderr-file  /Users/me/.opam/4.06.0/build/satysfi.~unknown/satysfi-38561-05296d.err
### stdout ###
# ruby ./gen_code.rb --gen-insttype src/frontend/bytecomp/vminstdef.yaml > src/frontend/__insttype.ml
### stderr ###
# ./gen_code.rb:166:in `gen_insttype': undefined method `load_documents' for Psych:Module (NoMethodError)
# Did you mean?  load_stream
# 	from ./gen_code.rb:254:in `call'
# 	from ./gen_code.rb:254:in `<main>'
# make: *** [src/frontend/types_.ml] Error 1

=-=- Error report -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫 
The following actions failed
  ∗  install satysfi ~unknown
No changes have been performed

environments

• macOS 10.113.4

$ opam switch
--     -- 3.07    Official 3.07 release
... [snip] ...
4.06.0  C 4.06.0  Official 4.06.0 release
--     -- 4.06.1  Official 4.06.1 release
system  I system  System compiler (4.06.0)
# 197 more patched or experimental compilers, use '--all' to show

$ opam --version
1.2.2

$ opam repository
  10 [git]   satysfi-external     https://github.com/gfngfn/satysfi-external-repo.git
   0 [http]     default     https://opam.ocaml.org

$ ruby -v
ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-darwin17]

$ gem list psych
*** LOCAL GEMS ***
psych (default: 3.0.2)

Travis CI for macOS always failed

I don't know why, but Travis CI jobs for macOS seems to be always errored, whereas jobs for Linux are passed. For example, look at this build.

The output says the compilation of ppx_derivers, cppo and result failed.
How about running cat stdout-file to know the reason of the errors?

lib-satysfi is copied to unexpected directory when `opam install satysfi`

Problem

It seems that opam install satysfi doesn't copy lib-satysfi to an appropriate place.

# After `opam install satysfi`...
SATySFi/tests$ make
satysfi list.satyh math.satyh head.satyh first.saty -o output.pdf
 ---- ---- ---- ----
  target file: 'output.pdf'
! [Error] package file not found: dist/hash/default-font.satysfi-hash
Makefile:4: recipe for target 'all' failed
make: *** [all] Error 1

Reason

This problem is related to a breaking change requested by #34. This PR changes the default location of the library to /usr/local/share/satysfi, but OPAM copies it under $HOME/.opam/4.06.0/share/satysfi because PREFIX=%prefix% while running opam install satysfi.

And, we shouldn't copy libraries under /usr/local/share while opam install, because the OPAM manual says:

(the installation commands) should only write to subdirectories of prefix, without altering the source directory itself.

In fact, copying to /usr/local/share requires sudo on Ubuntu.

Proposal

This problem will occur if non-default $PREFIX is set. How about making a config file which has all library path? I'll make a PR of this.

No, this proposal is the same as using ~/.satysfi. This causes a problem for Homebrew. Some other solution is needed (for example, using an environment variable like $LD_LIBRARY_PATH?).

Workaround

To newcomers: you can fix this issue temporarily by cp -R "$(opam config var prefix)/share/satysfi" "${HOME}/.satysfi".

Environment

• Ubuntu 17.10
• OCaml 4.06.0, OPAM 1.2.2
• SATySFi 3e6d067

Use opam repo instead of submodules

See: https://github.com/ocamllabs/advanced-fp-repo

A customized repo can be set up to have your customized camlpdf, otfm, and ucorelib there.

I can send some pull requests when I'm free.

A new syntax for string literals

I consider introducing a new syntax of the form

#`〜`, #``〜``, …

and

`〜`#, ``〜``#, …

not to ignore the leftmost/rightmost spaces in the string literals, respectively. Especially this will enable us to write the single space as #` `# instead of string-unexplode [0x0020].

Setup ocp-indent

Please consider using ocp-indent to enforce consistent indentation style.

Add instruction for WSL users

As reported in ocaml/opam-repository#12050 (comment), we have to disable bubblewrap to use OPAM on WSL. I believe that problem confuses many newbies who try to install SATySFi on WSL. (e.g., https://twitter.com/puripuri2100/status/1053254602273959936)

It’d better to have a notice or an instruction for WSL users.

Set version and release

Currently, there is no version number of SATySFi.
How about giving it a (semantic) version number?

Also, CHANGELOG would be needed.

`opam install satysfi` doesn't terminate (due to timeout of unicode.org)

Problem & Proposal

Sometimes, opam install satysfi doesn't terminate. In my environment, it seems that this is caused by the connection timeout of http://www.unicode.org, which hosts some Unicode data files (SATySFi downloads them in Makefile).

After some googling, I found that dodo/node-unicodetable#16 and dodo/node-unicodetable#19 say that http://www.unicode.org may throttle excess downloads. So opam install satysfi may fail when building SATySFi many times.

Therefore, how about using other ways to get UNIDATA files? There are some ways:

• Directly locate UNIDATA files in this repository
• Use UNIDATA files installed by the system (dodo/node-unicodetable partly do this)
• Download from mirror sites
• ...

I don't know which way is the best :(

How to reproduce

$ git clone --recursive https://github.com/gfngfn/SATySFi.git
$ cd SATySFi
$ make lib && make clean  # run this some times...
$ opam pin add satysfi .
$ opam install satysfi  # sometimes not terminate

Environment

• Ubuntu 17.10 (Artful Aardvark)
• OCaml 4.05.0 + OPAM 1.2.2
• SATySFi commit bc30f33

Workaround

For me, opam install satysfi succeeded after changing http://www.unicode.org/ in Makefile to ftp://ftp.unicode.org.

Interpreter Optimization

Background

SATySFi interpreter is implemented as a naïve AST traversal now. This is typically simple but slow. Now, contrary to expectations, we can find SATySFi documents doing complex computations such as peg, pi, and mandelbrot. They can also suffer from stack overflow (see #57), not only slowness.

Therefore, it's a good time to make SATySFi faster.

How to do it

Typically, the next step is to introduce a VM. Other options would be to introduce JIT or full compilation, but I don't think they're worth doing now.

Cannot infer types defined in modules

In Macrodown, one will be able to use modules like the code below (public type ... remains to be implemented):

module Mod = struct
  public type t = (int * string) list
  public let empty = []
  private let display-sub lst =
    match lst with
    | []             -> {}
    | (i, s) :: []   -> (arabic i) ^ {\ -> @s;}  
    | (i, s) :: tail -> (arabic i) ^ {\ -> @s;,\ } ^ (display-sub tail)
  public let display x = {[} ^ (display-sub x) ^ {]}
end-struct

Compared with OCaml, Macrodown does not have notion of signature, but has private, public and direct labels instead. However, this system makes the type inference system incomplete. For example, the compiler cannot infer from the code above which type display is intended to have, Mod.t -> string or (int * string) list -> string.

In order to avoid this nonuniqueness, Two choices are left:

1. Change the module system completely into the one based on signature. This is a fundamental solution for the problem, but perhaps makes the module system less convenient.
2. Make it mandatory to write annotations of types defined in modules explicitly. This choice relatively keeps the system practical, but distorts the omissibility of type annotations.

`cd tests ; make` fails due to missing files

Problem

Test files under test requires the following files which does not exist.

• imported.pdf
• naruse1.jpg

Furthermore, it requires several fonts which are not automatically installed. (It is related to #13)

• "KozMin": KozMinPro-Regular.otf
• "Hlv": HelveticaBlack.ttf
• "ipaexm": ipaexm.ttf
• "Osaka": Osaka.ttf
• "Arno": ArnoPro-Regular.otf
• "ArnoIt": ArnoPro-Italic.otf

Environment

• macOS 10.13.2
• OCaml 4.05.0 + OPAM 1.2.2
• SATySFi commit bc30f33

Workaround

• Create required PDF and JPEG files under test directory.
• Manually copy required fonts into lib-satysfi/dist/fonts directory. I can copy other fonts and modify lib-satysfi/dist/hash/fonts.satysfi-hash to change the correspondence between internal font names and actual font paths.

Feature request: use type constructors outside a module

Currently, we cannot construct a value of a variant type defined in a module, outside the module. I want to use these constructors.

Example:

module M = struct type t = A end

% We cannot use M.A here
% let x = M.A  % <-- ! [Syntax Error at Lexer]: illegal token '.' in a program area

% M.t itself can be used
let-rec forever () = forever ()
let x : M.t = forever ()  % pass the type checking

enable testing on CI

• Wait for Batteries' support for testing (ocaml-batteries-team/batteries-included#835)
• Replace uCoreLib (#40)
• Fix the font problem (#13)
• Fix the image problem for tests/first.saty
• Fix the imported.pdf problem (#17)
• Copy lib-satysfi to an appropriate place (#38)
• Install external fonts like IPAex Mincho (#17)
• Add build-test to the opam file

SATySFi says that all cross references were solved while they aren't

Consider the following mwe.saty. Note that \ref(`sec:fooo`); is a typo on purpose.

@require: stdja

StdJa.document (|
  title = {};
  author = {};
  show-title = false;
  show-toc = false
|) '<
  +section ?:(`sec:foo`) {Foo} <
    +p {
      This is Section \ref(`sec:fooo`);.
    }
  >
>

If we typeset this document, we obtain the log shown below and it claims all cross references were solved. after the 2nd trial.

$ satysfi mwe.saty -o mwe.pdf
 ---- ---- ---- ----
  target file: 'mwe.pdf'
  dump file: 'mwe.satysfi-aux' (will be created)
  parsing 'mwe.saty' ...
  parsing 'stdja.satyh' ...
  parsing 'pervasives.satyh' ...
  parsing 'gr.satyh' ...
  parsing 'list.satyh' ...
  parsing 'math.satyh' ...
  parsing 'color.satyh' ...
 ---- ---- ---- ----
  reading 'pervasives.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'list.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'color.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'math.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'gr.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'stdja.satyh' ...
  type check passed.
 ---- ---- ---- ----
  reading 'mwe.saty' ...
  type check: document
 ---- ---- ---- ----
  evaluating texts ...
  evaluation done.
 ---- ---- ---- ----
  breaking contents into pages ...
  needs another trial for solving cross references...
 ---- ---- ---- ----
  evaluating texts (2nd trial) ...
  evaluation done.
 ---- ---- ---- ----
  breaking contents into pages ...
  all cross references were solved.
 ---- ---- ---- ----
  embedding fonts ...
 ---- ---- ---- ----
  writing pages ...
 ---- ---- ---- ----
  output written on 'mwe.pdf'.

However, SATySFi obviously cannot find a label entry for sec:fooo and a question mark is printed on the output PDF. This situation wouldn't be considered as "solved" IMHO.