trueagi-io/metta-morph

Metta-morph (from Metamorphosis): Macro-based MeTTa to (Chicken) Scheme translator.

metta-morph

Metta-morph (from Metamorphosis): Macro-based MeTTa to (Chicken) Scheme translator.

The goal is to have a sufficient and fast (hundreds of times faster) subset of MeTTa implemented as an elegant Scheme library, rather than full language capability.

The thin Scheme compatibility layer is achieved via hygienic macros which utilize matching and ambivalence operations.

For a good overview, please also see Metta-morph presentation.

Installation

1. Install MeTTa interpreter: https://github.com/trueagi-io/hyperon-experimental

2. Install Chicken Scheme: https://www.call-cc.org/

3. Install the following chicken dependencies (install_dependencies.sh):

chicken-install srfi-69
chicken-install matchable
chicken-install amb

Run part of your code compiled from metta interpreter

In folder "extend":

compileme.metta:

(= (facF $n)
   (If (== $n 0)
       1
       (* $n (facF (- $n 1)))))

yourfile.metta:

!(extend-py! mettamorph)
!(compile! compileme.metta)
!(facF 42)

All functions defined in compile.me will automatically be compiled and callable, whereby on next run it will not be compiled again until changes to compileme.metta are made.

Another option is to compile code in-line:

yourfile.metta:

!(extend-py! mettamorph)
!(compile! "
(= (facF $n)
   (If (== $n 0)
       1
       (* $n (facF (- $n 1)))))
")
!(facF 42)

In both cases simply run with

metta yourfile.metta

Run code file with interpreter

sh run.sh filename.metta

which will produce RUN.scm and RUN.metta from filename.metta and run them with the MeTTa interpreter and Chicken Scheme interpreter. The output will be timed, and compared with each other, showing either == or !=. If != is shown then the recommended practice is to adjust the file appropriately with limitations in mind, until it will show with ==. Hence run.sh is a workable tool to make sure your code is independent from the particular interpreter implementation.

Compile code file to binary

sh compilescheme.sh

which will build RUN.scm into a binary RUN one can execute with ./RUN or time ./RUN.

Run tests

sh test.sh

The file name of tests which led to different outputs with MeTTa and Chicken Scheme get printed out.

Current limitations

• When a variable from the outer scope is again used in case and let statements, in MeTTa it will be constrained to have the same value as in the outer scope while in Chicken Scheme the outer variable will be shadowed within the local context. Do not introduce same variable names again in inner scope and both will behave the same. If equality constraint is intended, just use an explicit If statement.

• Basic type annotations are mostly MeTTa-compatible and used to generate more performant code, but the type system is not fully compatible. With Mettamorph as extension library the MeTTa type system can be fully used.

• Partial evaluation, e.g. leaving variables as variables when calling a function is not supported.

• Mettamorph has its own &self space, but as shown in the examples, interop is supported with wrapper functions.

These limitations are relatively minor, a substantial part of MeTTa is supported. Most importantly, the toolset allows to ensure compatibility with the MeTTa interpreter is incrementally preserved during development and the compiled code can easily be called with the MeTTa interpreter. Compared to the MeTTa interpreter the code executes by a factor of 10x+ faster with the Chicken Scheme interpreter, and usually 200x+ faster with the compiler (please see the timing folder for objective quantification).

Complex example

The most complex example to date is running full metta-nars (not just minnars.metta and nalifier.metta which is part of the test suite) with Mettamorph. https://github.com/patham9/metta-nars

metta-morph's Issues

Parallel superpose

An attempt I tried a few weeks ago which worked out and I forgot to document (OS-process-based ambivalence to let one easily make use of CPU resources)

(import (chicken process) (chicken string) (chicken file posix) amb)

(define-syntax hyperpose
  (syntax-rules ()
    ((_ arg1 arg2)
     (receive (pipefd0 pipefd1) (create-pipe)
              (let ((pid (process-fork)))
                   (if (eq? pid 0)
                       (begin (write arg1 (open-output-file* pipefd1))
                              (exit 0)) ;child is done
                       (let* ((res1 arg2) ;get to work before waiting for child to return result
                              (res2 (read (open-input-file* pipefd0))))
                             (amb res1 res2))))))))

;example:
(define (test x) (list 2 x (+ 2 x)))
(display (amb-collect (hyperpose (test 1) (test 2))))
(newline)

This is just an example, forking has some time overhead so use only if you have computationally expensive branches which should be able to be explored with different CPU cores!

Quantifying the speedup

./timing/timing.py should collect and report average and variances across multiple runs rather than time of a single run.

Most recent output:
https://gist.github.com/patham9/9af1fb750ed37dd3b0c31c3740e7ccd9

Presentation file

MeTTa-morph.pdf
(updated link)

Support MeTTa expressions instead of code strings in compile! function

As Vitaly and Alexey pointed out, it would be good if !(compile ...) would accept a MeTTa expression instead of a code string. That way it becomes possible to derive program code at runtime and then to pass it to the compiler.

Example, current:

!(compile! "
(= (facF $n)
   (If (== $n 0)
       1
       (* $n (facF (- $n 1)))))
")

Example desired:

!(compile! 
(= (facF $n)
   (If (== $n 0)
       1
       (* $n (facF (- $n 1)))))
)

Hereby, the optional possibility to compile a MeTTa file can be preserved:
!(compile! test.metta)

Allow for multiple invocations of compile! rather than a single one

Currently only a single compile! call is supported, though can include multiple functions within the compiled code.

It should however be possible to just build and load additional .so files, the challenge though is that each references the Chicken Scheme .so files, which it seems can only be referenced once in a process. There is likely a way around this though!

The choice of Scheme compiler

Chicken Scheme looks ok, but other choices can be considered carefully in order to proceed further. There are other choices such as Dr.Racket or Guile with different approaches (e.g. compilation to a VM rather than to a shared library). We need to create a list of suitable Scheme compilers and analyze them in accordance with a number of properties, e.g.

• speed of execution of compiled code (can be measured without integrating to MeTTa)
• speed of compilation
• flawless integration to MeTTa, i.e. will MeTTa code with compiled functions and grounded atoms be executable (like it is done for Chicken Scheme through textual representations and eval), or are there pitfalls
• platform support and easiness of deployment (in particular, possibility to install by metta-morph itself without user actions; or possibility to install as a Python package)
• availability of libraries and possibility to install them and make accessible from MeTTa without user actions
• availability of API (besides CLI) for compilation either in Scheme itself or in one of C++/Rust/Python
• compilation control and other features: compilation to shared libraries or/and RAM, detection of the necessity to recompile stuff between different runs, compilation error handling, etc.
• support for type-checking [optional?]
• ...

Tracking issue: repo structure

The repo structure cannot be fleshed out completely until packaging mechanisms are introduced to MeTTa, but some steps might be doable now. This issue serves as a place, where we can discuss if child issues for particular changes should be created or not.

Testcase Syntax

Going to suggest for testcases (since this works in Rust MeTTa) we adopt a standard test writing format

; it is non-deterministic: each value is matched against all cases
!(assertEqual
 (case (+ 1 (superpose (1 2 3)))
  ((3 OK-3)
   (4 OK-4)))
 (superpose (OK-3 OK-4)))

; one case can produce multiple results
!(assertEqual
  (case (+ 1 (superpose (1 2 3)))
     (($x (+ 1 $x))))
  (superpose (3 4 5)))

Introduce unit tests for branch protection

Unit tests should systematically cover various aspects of MeTTa-Scheme interactions and run automatically for branch protection on github. Not working cases could possibly exist as skipped unit tests. More complex examples can remain as integration tests.

Project/library name

metta-morph sounds nice, and I don't argue for changing its name, but we may need to just fix the decision.

• As for metta- prefix: import metta-something in MeTTa looks redundant. However, it's ok to have it for the repo or even package name. E.g., pytorch is a package name, while it is imported as torch. MeTTa package installation is not ready, so it is more a question about the repo name. metta- sounds suitable for it.

• As for morph, it is not too discriminative and descriptive. It says nothing about Scheme or compilation. However, it may become a library for transforming metta programs (in a meta-programming style), who knows. Thus, it is not completely out of place.

In total, metta-morph is OK, but if it was considered as a preliminary working name and there are any ideas for renaming, then it is timely to consider them or decide to stay with the current name.