mooreryan/ocaml_python_bindgen

Generate Python bindings via pyml from OCaml value specifications

OCaml-Python Bindings Generator

Generate Python bindings with pyml directly from OCaml value specifications.

While you could write all your Python bindings by hand, it can be tedious and it gets old real quick. While pyml_bindgen can't yet auto-generate all the bindings you may need, it can definitely take care of a lot of the tedious and repetitive work you need to do when writing bindings for a big Python library!! 💖

Quick start

First, install pyml_bindgen. It is available on Opam.

$ opam install pyml_bindgen

Say you have a Python class you want to bind and use in OCaml. (Filename: adder.py)

class Adder:
    @staticmethod
    def add(x, y):
        return x + y

To do so, you write OCaml value specifications for the class and methods you want to bind. (Filename: val_specs.txt)

val add : x:int -> y:int -> unit -> int

Then, you run pyml_bindgen.

$ pyml_bindgen val_specs.txt adder Adder --caml-module Adder > lib.ml

Now you can use your generated functions in your OCaml code. (Filename: run.ml)

open Lib

let () = Py.initialize ()

let result = Adder.add ~x:1 ~y:2 ()

let () = assert (result = 3)

Finally, set up a dune file and run it.

(executable
 (name run)
 (libraries pyml))

$ dune exec ./run.exe

Documentation

For information on installing and using pyml_bindgen, check out the docs.

Additionally, you can find examples in the examples directory. One neat thing about these examples is that you can see how to write Dune rules to automatically generate your pyml bindings.

You may also want to check out my blog post introducing pyml_bindgen.

Installing from sources

If you want to install from sources, e.g., to track the main branch or a development branch, but you do not want to install all the test and development packages, clone the repository, checkout the branch you want to follow and install:

$ git clone https://github.com/mooreryan/ocaml_python_bindgen.git
# Checkout whatever branch you want, in this case `dev`.
$ git checkout dev
$ make opam_install

This will save a lot of install time as it avoids some heavy packages.

Development

If instead, you want to work on pyml_bindgen development, will need to ensure you have the development and test dependencies.

E.g.,

$ git clone https://github.com/mooreryan/ocaml_python_bindgen.git
$ make deps
$ make deps_dev
# Start working!
$ dune test

License

Copyright (c) 2021 - 2022 Ryan M. Moore.

Licensed under the Apache License, Version 2.0 or the MIT license, at your option. This program may not be copied, modified, or distributed except according to those terms.

ocaml_python_bindgen's Issues

embedding more than one python file

Just an idea.
Currently, I have one python file that I write to interface w/ ocaml (let's call it itf.py).
I.e. the ocaml side only calls functions defined in this file.
However, this said file may rely on other python files in the same directory to implement
complex things.

It would be nice if we can embed in some way those other files too.

Currently, I had to include by hand the other files into itf.py.
But, I hate to do this, I just duplicated code, if there are some bugs, I will need to correct in two places from now on.

Do you understand my need?

Would it be doable?

I could (should?) create a python library containing all those other files.
But then, the ocaml exe will rely on this library to be installed at runtime.
In terms of deployment, it would be nicer is the ocaml exe can embed everything needed (i.e. itf.py plus
all its required python files dependencies).

Generated OCaml interface's style

It would be nice if people don't have to put this penultimate unit
at the end of a function declaration (in the *_specs.txt file).
Once you are in ocaml, having to add those () to call a function
doesn't look good.
But, I admit it's just a question of style.

there are many depends in the opam file

I don't know if you can or even should do something about it.
But, it makes the install quite slow; also, I suspect the install will break easily in the future, or be incompatible with what the user already have installed.

adder.py needs to be in the current working directory

or you get this kind of error:

Fatal error: exception E (<class 'ModuleNotFoundError'>, No module named 'adder')

Is there a way I can install this python file somewhere else, but so that
it could still be found at runtime?

t option and custom option arguments are tricky to handle correctly

So this is sort of a subtly weird thing that can trip you up if you're not careful.

Say you have a python fun like this:

    def say_hi(self, to_whom=None, greeting='hi'):
        if to_whom:
            return(f'{self.name} says {greeting} to {to_whom.name}')
        else:
            return(f'{self.name} says {greeting}')

Given that to_whom takes None in the python, you decide to make it explicit in the OCaml binding.

  val say_hi : t -> to_whom:t option -> greeting:string option -> unit -> string

However, this generates code that has the wrong signature

  let say_hi t ~to_whom ~greeting () =
    let callable = Py.Object.find_attr_string t "say_hi" in
    let kwargs =
      filter_opt
        [
          Some ("to_whom", to_pyobject to_whom);
          Some
            ( "greeting",
              (function Some x -> Py.String.of_string x | None -> Py.none)
                greeting );
        ]
    in
    Py.String.to_string
    @@ Py.Callable.to_function_with_keywords callable [||] kwargs

That sig is val say_hi : t -> to_whom:t -> greeting:string option -> unit -> string.

To make that arg actually optional with t, you would have to do this instead.

  val say_hi2 : t -> ?to_whom:t -> greeting:string option -> unit -> string

Note that this problem is only there for t option and custom option eg Doc.t option.

Fixing it

It's not straightforward to fix as t option as an argument passed to a python function and t option as a return type of an OCaml function need to be treated differently (and that depends on the --of-pyo-ret-type flag).

Short term fix is to document the weirdness and give an error when users try to use t option in this way. Currently, the bindings will be generated and you won't know something is wrong until you try to build it.

Long term is to actually handle t option properly as an arg and also properly as a return type.

pyml_bindgen invocation example in case there is no class [question]

Do you have a command-line example in case you generate bindings for a python
file with a single function inside, no class defined?

supporting python functions returning tuples would be _very_ useful

in the *_specs.txt file.
At first, maybe supporting only tuples where all elements are of the same type would be nice.

For example, support:

int * int
int * int * int

up to the same arity pyml supports.
I needed to wrap a Python class where the returned values are:
(int * int * int * int * int) list
and
(int * int * int) list

--embed-python-source

Hello Ryan,

@thierry-martinez just told us how to embed some python source code
directly into the ocaml source:
thierry-martinez/pyml#78

I gave it a try and it "just works".
It would be nice if pyml_bindgen had an option to enable this when generating some ocaml bindings.

Currently, you generate a line like this in the generated .ml file:

  let import_module () =
    Py.Import.import_module "python_module_name"

With the utility function Thierry gave us, this generated line should be replaced by:

  let import_module () =
    import_python_source
      ~module_name:"toto" ~filename:"toto.py"
    ~source:{|
<the content of toto.py>
|}

Thanks to this, at run-time we don't need toto.py to have been installed by the user / be in PYTHONPATH.

If you implement this and want me to help testing the feature, just let me know.

You don't always need to generate the pyobject conversions functions

If you generate with the option --associated-with module then the resulting OCaml module shouldn't need a type t or the of_pyo and to_pyo functions. But currently they are generated.

Look in to whether we can drop some of the extra functions in certain situations like these.

"could not create val_spec" is not very helpful

When one is missing the penultimate unit parameter in the *_specs.txt file,
this is the error message to get.
There could be a more helpful message, or at least that would be nice.