papierkorb/bindgen

Binding and wrapper generator for C/C++ libraries

bindgen's Issues

Simplify find_clang.cr

There is a bit of overlap between clang/find_clang.cr and clang/CMakeLists.txt.

find_clang supports finding the clang++ binary, but this is redundant as CMake will figure it out and call find_clang with option --clang PATH.

Also, as @kalinon mentions in #32 , probably more optimizations can be made by directly reading values from one of:

llvm-config
    --cxxflags
    --ldflags
    --libs all
    --includedir

Instead of parsing those values out manually from parts of clang++ output.

Better cxxflags setting in spec/clang/spec_helper.cr

spec_helper.cr needs access to the cxxflags variable which is determined when the clang tool (subdir clang/ runs).
Currently, the tool dumps variables to Makefile.variables, in Makefile format, and spec_helper reads it in a basic way.

diff --git a/spec/clang/spec_helper.cr b/spec/clang/spec_helper.cr
index 71e0bd0..672c502 100644
--- a/spec/clang/spec_helper.cr
+++ b/spec/clang/spec_helper.cr
@@ -18,8 +18,17 @@ def clang_tool(cpp_code, arguments, **checks)

   tool = ENV["BINDGEN_BIN"]? || Bindgen::Parser::Runner::BINARY_PATH

+  cxx_flags = "-std=c++11 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS"
+  makefile_vars = File.join(__DIR__, "../../clang/Makefile.variables")
+  if File.exists?(makefile_vars)
+    File.read_lines(makefile_vars).each do |line|
+      if line =~ /^LLVM_CXX_FLAGS/
+        cxx_flags = line.split(" := ").last.chomp
+      end
+    end
+  end
   command = "#{tool} #{file.path} #{arguments} -- " \
-            "-x c++ -std=c++11 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS " \
+            "-x c++ #{cxx_flags} " \
             "-Wno-implicitly-unsigned-literal"

We should change this to pass this data in a more formal way. One possible way to do it would be for the clang tool to read some (existing or new) file in YAML or Crystal format.

Current Broken Specs and Issues

@docelic I noticed we were both fixing some of the same stuff, so here is what i have left and what ive found.

Broken specs:

crystal spec spec/integration/containers_spec.cr:4 # container instantiation feature works
crystal spec spec/integration/arguments_spec.cr:4 # the argument translation functionality works

arguments_spec:

in [spec/integration/arguments.cpp] we have a test for setting the default string:

  int defaultString(std::string str = "Okay") {
    return str.length();
  }

However bindgen is currently outputting the following JSON when parsing it:

{
          "type": "MemberMethod",
          "access": "Public",
          "name": "defaultString",
          "isConst": false,
          "isVirtual": false,
          "isPure": false,
          "isExternC": false,
          "className": "Defaults",
          "firstDefaultArgument": 0,
          "arguments": [
            {
              "isConst": false,
              "isMove": false,
              "isReference": false,
              "isBuiltin": false,
              "isVoid": false,
              "pointer": 0,
              "baseName": "std::string",
              "fullName": "std::string",
              "template": {
                "baseName": "std::__1::basic_string",
                "fullName": "std::basic_string<char, std::char_traits<char>, std::allocator<char> >",
                "arguments": [
                  {
                    "isConst": false,
                    "isMove": false,
                    "isReference": false,
                    "isBuiltin": true,
                    "isVoid": false,
                    "pointer": 0,
                    "baseName": "char",
                    "fullName": "char",
                    "template": null
                  },
                  {
                    "isConst": false,
                    "isMove": false,
                    "isReference": false,
                    "isBuiltin": false,
                    "isVoid": false,
                    "pointer": 0,
                    "baseName": "std::char_traits<char>",
                    "fullName": "std::char_traits<char>",
                    "template": {
                      "baseName": "std::__1::char_traits",
                      "fullName": "std::char_traits<char>",
                      "arguments": [
                        {
                          "isConst": false,
                          "isMove": false,
                          "isReference": false,
                          "isBuiltin": true,
                          "isVoid": false,
                          "pointer": 0,
                          "baseName": "char",
                          "fullName": "char",
                          "template": null
                        }
                      ]
                    }
                  },
                  {
                    "isConst": false,
                    "isMove": false,
                    "isReference": false,
                    "isBuiltin": false,
                    "isVoid": false,
                    "pointer": 0,
                    "baseName": "std::allocator<char>",
                    "fullName": "std::allocator<char>",
                    "template": {
                      "baseName": "std::__1::allocator",
                      "fullName": "std::allocator<char>",
                      "arguments": [
                        {
                          "isConst": false,
                          "isMove": false,
                          "isReference": false,
                          "isBuiltin": true,
                          "isVoid": false,
                          "pointer": 0,
                          "baseName": "char",
                          "fullName": "char",
                          "template": null
                        }
                      ]
                    }
                  }
                ]
              },
              "hasDefault": true,
              "isVariadic": false,
              "name": "str",
              "value": null
            }
          ],
          "returnType": {
            "isConst": false,
            "isMove": false,
            "isReference": false,
            "isBuiltin": true,
            "isVoid": false,
            "pointer": 0,
            "baseName": "int",
            "fullName": "int",
            "template": null
          }
        }

Notice in particular:

              "hasDefault": true,
              "isVariadic": false,
              "name": "str",
              "value": null

So while it is noticing that there is a default value, the value itself is being returned as null.
This is happening in [clang/src/type_helper.cpp]

Argument TypeHelper::processFunctionParameter(const clang::ParmVarDecl *decl) {
	clang::ASTContext &ctx = decl->getASTContext();
	Argument arg;

	clang::QualType qt = decl->getType();
	qualTypeToType(arg, qt, ctx);
	arg.name = decl->getQualifiedNameAsString();
	arg.isVariadic = false;
	arg.hasDefault = decl->hasDefaultArg();
	arg.value = JsonStream::Null;

	// If the parameter has a default value, try to figure out this value.  Can
	// fail if e.g. the call has side-effects (Like calling another method).  Will
	// work for constant expressions though, like `true` or `3 + 5`.
	if (arg.hasDefault) {
		TypeHelper::readValue(arg.value, qt, ctx, decl->getDefaultArg());
	}

	return arg;
}

and paricularly in:

bool TypeHelper::valueFromApValue(LiteralData &value, const clang::APValue &apValue, const clang::QualType &qt) {
	if (qt->isPointerType()) {
		// For a pointer-type, just store if it was `nullptr` (== true).
		value = apValue.isNullPointer();
	} else if (qt->isBooleanType()) {

Where it sets it as a nullptr

Clang 10 support

You can cherrypick my commit, just change macros to the right one: ZaWertun@125f32c

Problems with `std::vector`... and more

I'm trying to use this to create bindings for this project. https://github.com/openSUSE/libstorage-ng

But I'm having problems when it comes to collections. I can't find the right incantations I must use in TEMPLATE.yml to make it work.

To illustrate the problems, I have created a repository with a super-small subset of libstorage-ng (only one struct, one function and one enum). The README in that repo should explain the problem. Please see https://github.com/ancorgs/bindgen-test

Qt-5.11: Bindgen generates binding with duplicate values in enum Qt::Key

Here is part of enum Qt::Key that causing compilation errors (enum 'Qt::Key' already contains a member named 'DeadA'):

enum Key : UInt32
...
    DeadA = 16781952
    DeadA = 16781953
    DeadE = 16781954
    DeadE = 16781955
    DeadI = 16781956
    DeadI = 16781957
    DeadO = 16781958
    DeadO = 16781959
    DeadU = 16781960
    DeadU = 16781961
...
end

Bug caused by new values for enum Key entroduced in Qt 5.11:

enum Key {
...
    Key_Dead_a              = 0x01001280,
    Key_Dead_A              = 0x01001281,
    Key_Dead_e              = 0x01001282,
    Key_Dead_E              = 0x01001283,
    Key_Dead_i              = 0x01001284,
    Key_Dead_I              = 0x01001285,
    Key_Dead_o              = 0x01001286,
    Key_Dead_O              = 0x01001287,
    Key_Dead_u              = 0x01001288,
    Key_Dead_U              = 0x01001289,
...
}

After camelcase translation both Dead_a and Dead_A becomes DeadA 😕

bindgen.cpp:1:10: fatal error: 'clang/Tooling/CommonOptionsParser.h' file not found

This is what I get when I try to run crystal deps with Qt5.cr, with the master-ready-to-use branch.

g++ -o bindgen bindgen.cpp -std=c++11 -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -lclangFrontend-lclangSerialization -lclangDriver -lclangTooling -lclangToolingCore -lclangParse -lclangRewriteFrontend -lclangStaticAnalyzerFrontend -lclangSema -lclangAnalysis -lclangEdit -lclangAST -lclangLex -lclangBasic  -lclangASTMatchers -lLLVMX86AsmParser -lLLVMX86Desc -lLLVMX86AsmPrinter -lLLVMX86Info -lLLVMX86Utils -lLLVMipo -lLLVMScalarOpts -lLLVMInstCombine -lLLVMTransformUtils -lLLVMAnalysis -lLLVMTarget -lLLVMOption -lLLVMMCParser -lLLVMMC -lLLVMObject -lLLVMBitReader  -lLLVMCore -lLLVMProfileData -lLLVMSupport -lLLVMDemangle -ldl -pthread -lz -lcurses
bindgen.cpp:1:10: fatal error: 'clang/Tooling/CommonOptionsParser.h' file not found
#include "clang/Tooling/CommonOptionsParser.h"

Better fix for unsafe_fetch argument type

Creating the issue to keep track of a better/cleaner equivalent of commit:

4012779

The issue itself, for which this is a patch, is described/mentioned in #26 .

Fix builds for 0.7.0

Currently the builds are broken: https://travis-ci.org/github/Papierkorb/bindgen

Some namespaced class hierarchies are not realizable

In Crystal every class must be defined after all of its base classes, but #83 now permits certain class hierarchies that cannot be realized by the current Bindgen generators, because they require reopening a namespace:

namespace N {
  struct A { };
}

struct B : N::A { };

namespace N {
  struct C : B { };
}

# `N::A` is not defined here
class B < N::A end

module N
  class A end
  class C < B end
end

module N
  class A end
  # `B` is not defined here
  class C < B end
end

class B < N::A end

This rarely occurs in real code; a full fix might require a complete rewrite of Graph::Container and friends, since they impose similar restrictions on the node visitation order.

clang/find_clang.cr can't find installed clang on Fedora 28

Though it works after this patch:

diff --git a/clang/find_clang.cr b/clang/find_clang.cr
index d7d4d61..c86f9f6 100644
--- a/clang/find_clang.cr
+++ b/clang/find_clang.cr
@@ -150,10 +150,17 @@ end
 # Find all LLVM and clang libraries, and link to all of them.  We don't need
 # all of them - Which totally helps with keeping linking times low.
 def find_libraries(paths, prefix)
+  res = [] of String
   paths
-    .flat_map{|path| Dir["#{path}/lib#{prefix}*.a"]}
-    .map{|path| File.basename(path)[/^lib([^.]+)\.a$/, 1]}
+    .map{|p| File.expand_path(p)}
     .uniq
+    .flat_map{|path| Dir["#{path}/lib#{prefix}*.so"]}
+    .each do |path|
+      if File.basename(path) =~ /^lib([^.]+)\.so$/
+        res << $1
+      end
+    end
+  res.uniq
 end
 
 llvm_libs = find_libraries(system_libs, "LLVM")
@@ -166,7 +173,7 @@ print_help_and_bail if llvm_libs.empty? || clang_libs.empty?
 # into the compiler, we ensure this.  The probably laziest dependency resolution
 # algorithm in existence.
 libs = (clang_libs + clang_libs + llvm_libs + llvm_libs).map{|x| "-l#{x}"}
-includes = system_includes.map{|x| "-I#{x}"}
+includes = system_includes.map{|p| File.expand_path(p)}.map{|x| "-I#{x}"}
 
 puts "CLANG_LIBS := " + libs.join(" ")
 puts "CLANG_INCLUDES := " + includes.join(" ")

Also I've got strange error from crystal when executing File.basename(path)[/^lib([^.]+)\.a$/, 1], something about null reference, so I rewrited it with:

if File.basename(path) =~ /^lib([^.]+)\.so$/
    res << $1
end

Extra path processing with File.expand_path eliminates horrible paths with lot of ../../../ inside.

It would be nice if you test this patch on systems other than Fedora.

Rebuilding Qt bindings for upcoming Crystal 1.0 release

Hey @Papierkorb , @HertzDevil , it would be really nice if we planned to rebuild the Qt bindings for various versions roughly in sync with Crystal's 1.0 release.

@HertzDevil do you have a general overview of the things that you still want to add to bindgen, which directly influence the Qt bindings?
And/or @Papierkorb, do you have a list of areas in bindgen that still need improvement in relation to Qt?

Thanks!

Public instance properties

My current focus is getting QStyleOption and its subclasses to work. However, these classes use public instance variables, and Bindgen doesn't expose them to Crystal wrapper classes:

class QStyleOption
{
public:
    int version;
    int type;
    QStyle::State state;
    Qt::LayoutDirection direction;
    QRect rect;
    QFontMetrics fontMetrics;
    QPalette palette;
    QObject *styleObject;

    QStyleOption(int version = QStyleOption::Version, int type = SO_Default);
    void init(const QWidget *w);
    inline void initFrom(const QWidget *w) { init(w); }
    // ...
};

module Qt
  class StyleOption
    def initialize(version : Int32 = 1, type : Int32 = 0)
      # ...
    end
    
    def init(w : Widget) : Void
      # ...
    end
    
    def init_from(w : Widget) : Void
      # ...
    end

    # facilities for @unwrap, nothing else
  end
end

My plan is to add a new processor that synthesizes getter / setter methods from public instance variables, with binding function names like version_GETTER and version_SETTER:

extern "C" int bg_QStyleOption_version_GETTER_(QStyleOption * _self_) {
  return _self_->version;
}

extern "C" void bg_QStyleOption_version_SETTER_int(QStyleOption * _self_, int version) {
  _self_->version = version;
}

module Qt
  lib Binding
    fun bg_QStyleOption_version_GETTER_(_self_ : QStyleOption*) : Int32
    fun bg_QStyleOption_version_SETTER_int(_self_ : QStyleOption*, version : Int32) : Void
  end

  class StyleOption
    def version : Int32
      Binding.bg_QStyleOption_version_GETTER_(self)
    end

    def version=(version : Int32) : Void
      Binding.bg_QStyleOption_version_SETTER_int(self, version)
    end
  end
end

The alternative is to mirror the whole class hierarchy to lib structs, but I feel like this creates more problems than it is worth. (Qt doesn't treat these objects as values either.)

LLVM 7 - PIE / fPIC

On Debian and LLVM 7, after bindgen is compiled, Qt specs fail with:

>>> Output of failed spec: virtual_override.yml
/usr/bin/ld: /home/x/bindgen/spec/integration/tmp/../tmp/virtual_override.o: relocation 
R_X86_64_32S against symbol `stderr@@GLIBC_2.2.5' can not be used when making a PIE 
object; recompile with -fPIC
/usr/bin/ld: final link failed: nonrepresentable section on output
collect2: error: ld returned 1 exit status

>>> Output of failed spec: c_wrapper.yml
/usr/bin/ld: /home/x/bindgen/spec/integration/tmp/../tmp/c_wrapper.o: relocation R_X86_64_32S 
against `.bss' can not be used when making a PIE object; recompile with -fPIC

>>> Output of failed spec: containers.yml
/usr/bin/ld: /home/x/bindgen/spec/integration/tmp/../tmp/containers.o: relocation R_X86_64_32 
against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC

>>> Output of failed spec: basic.yml
/usr/bin/ld: /home/x/bindgen/spec/integration/tmp/../tmp/basic.o: relocation R_X86_64_32 against 
symbol `__gxx_personality_v0@@CXXABI_1.3' can not be used when making a PIE object; 
recompile with -fPIC

>>> Output of failed spec: qt.yml
/usr/bin/ld: /home/x/bindgen/spec/integration/tmp/../tmp/qt.o: relocation R_X86_64_32S against 
symbol `_ZN10SomeObject13stuffHappenedEv' can not be used when making a PIE object; 
recompile with -fPIC

An example of a CC line that's invoked is:

Error: execution of command failed with code: 1: `cc "${@}" -o '/home/x/.cache/crystal/crystal-run-
qt_test.tmp' -lgccpp -rdynamic  /home/x/bindgen/spec/integration/tmp/../tmp/qt.o -lstdc++ -lpcre 
-lm /opt/crystal-0.34.0-1/bin/../lib/crystal/lib/libgc.a -lpthread /opt/crystal-0.34.0-1/share/crystal
/src/ext/libcrystal.a -levent -lrt -ldl -L/opt/crystal-0.34.0-1/bin/../lib/crystal/lib -L/opt/crystal-0.34.0-1
/bin/../lib/crystal/lib`

`find_clang` fails to find libraries

When running find_clang.cr, it fails to find library paths, unless I add p expression:

def find_libraries(paths, prefix, dynamic=false)
  if dynamic
    paths
      .flat_map { |path| Dir["#{path}/lib#{prefix}*.so"] }
      .map { |path| File.basename(path)[/^lib(.+)\.so$/, 1] }
      .uniq
      .to_a

    p paths # <- works with this
  else
    paths
      .flat_map { |path| Dir["#{path}/lib#{prefix}*.a"] }
      .map { |path| File.basename(path)[/^lib([^.]+)\.a$/, 1] } # FIXME: this lead to crash for e.g. libclang_rt.msan_cxx-x86_64.a
      .uniq
      .to_a

    p paths # <- works with this
  end
end

Clang environment:

clang version 10.0.0 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
 "/usr/bin/clang-10" "-cc1" "-triple" "x86_64-pc-linux-gnu" "-emit-obj" "-mrelax-all" "-disable-free" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "bindgen.cpp" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-mthread-model" "posix" "-mframe-pointer=all" "-fmath-errno" "-fno-rounding-math" "-masm-verbose" "-mconstructor-aliases" "-munwind-tables" "-target-cpu" "x86-64" "-dwarf-column-info" "-fno-split-dwarf-inlining" "-debugger-tuning=gdb" "-resource-dir" "/usr/lib/clang/10.0.0" "-internal-isystem" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0" "-internal-isystem" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/x86_64-pc-linux-gnu" "-internal-isystem" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/backward" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/lib/clang/10.0.0/include" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-fdeprecated-macro" "-fdebug-compilation-dir" "/home/kmeinkopf/CodeProjects/crystal/btorrent-cr/lib/bindgen/clang" "-ferror-limit" "19" "-fmessage-length" "0" "-stack-protector" "2" "-fgnuc-version=4.2.1" "-fobjc-runtime=gcc" "-fcxx-exceptions" "-fexceptions" "-fdiagnostics-show-option" "-fcolor-diagnostics" "-faddrsig" "-o" "/tmp/bindgen-0618b1.o" "-x" "c++" "/home/kmeinkopf/CodeProjects/crystal/btorrent-cr/lib/bindgen/clang/src/bindgen.cpp"
 "/usr/bin/ld" "-pie" "--eh-frame-hdr" "-m" "elf_x86_64" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../lib64/Scrt1.o" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../lib64/crti.o" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/crtbeginS.o" "-L/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0" "-L/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../lib64" "-L/usr/bin/../lib64" "-L/lib/../lib64" "-L/usr/lib/../lib64" "-L/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../.." "-L/usr/bin/../lib" "-L/lib" "-L/usr/lib" "/tmp/bindgen-0618b1.o" "-lstdc++" "-lm" "-lgcc_s" "-lgcc" "-lc" "-lgcc_s" "-lgcc" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/crtendS.o" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../lib64/crtn.o"
╭─kmeinkopf@karl-pc ~/CodeProjects/crystal/btorrent-cr/lib/bindgen/clang ‹master*› 
╰─$ /usr/bin/clang++ -### /home/kmeinkopf/CodeProjects/crystal/btorrent-cr/lib/bindgen/clang/src/bindgen.cpp
clang version 10.0.0 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
 "/usr/bin/clang-10" "-cc1" "-triple" "x86_64-pc-linux-gnu" "-emit-obj" "-mrelax-all" "-disable-free" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "bindgen.cpp" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-mthread-model" "posix" "-mframe-pointer=all" "-fmath-errno" "-fno-rounding-math" "-masm-verbose" "-mconstructor-aliases" "-munwind-tables" "-target-cpu" "x86-64" "-dwarf-column-info" "-fno-split-dwarf-inlining" "-debugger-tuning=gdb" "-resource-dir" "/usr/lib/clang/10.0.0" "-internal-isystem" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0" "-internal-isystem" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/x86_64-pc-linux-gnu" "-internal-isystem" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/backward" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/lib/clang/10.0.0/include" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-fdeprecated-macro" "-fdebug-compilation-dir" "/home/kmeinkopf/CodeProjects/crystal/btorrent-cr/lib/bindgen/clang" "-ferror-limit" "19" "-fmessage-length" "0" "-stack-protector" "2" "-fgnuc-version=4.2.1" "-fobjc-runtime=gcc" "-fcxx-exceptions" "-fexceptions" "-fdiagnostics-show-option" "-fcolor-diagnostics" "-faddrsig" "-o" "/tmp/bindgen-14a2ba.o" "-x" "c++" "/home/kmeinkopf/CodeProjects/crystal/btorrent-cr/lib/bindgen/clang/src/bindgen.cpp"
 "/usr/bin/ld" "-pie" "--eh-frame-hdr" "-m" "elf_x86_64" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../lib64/Scrt1.o" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../lib64/crti.o" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/crtbeginS.o" "-L/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0" "-L/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../lib64" "-L/usr/bin/../lib64" "-L/lib/../lib64" "-L/usr/lib/../lib64" "-L/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../.." "-L/usr/bin/../lib" "-L/lib" "-L/usr/lib" "/tmp/bindgen-14a2ba.o" "-lstdc++" "-lm" "-lgcc_s" "-lgcc" "-lc" "-lgcc_s" "-lgcc" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/crtendS.o" "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../lib64/crtn.o"

OS:

DISTRIB_ID=ManjaroLinux
DISTRIB_RELEASE=20.0.3
DISTRIB_CODENAME=Lysia
DISTRIB_DESCRIPTION="Manjaro Linux"

find_clang.cr finds different versions of clang & llvm-config

There maybe several clang & llvm versions installed on user computer.
For example I have both llvm/clang 6 and 10.
After running find_clang.cr I've got this variables (from Makefile.variables):

CLANG_BINARY := /bin/clang++-10
CLANG_INCLUDES := -I/usr/include -I/include/c++/10
CLANG_LIBS := -Wl,--start-group -lclang -lclang-cpp -Wl,--start-group -Wl,--start-group -lLLVM-10 -lLLVM-10.0.0 -lLLVM -Wl,--start-group
LLVM_CONFIG_BINARY := /bin/llvm-config-6.0-64
LLVM_VERSION := 6
LLVM_VERSION_FULL := 6.0.1
LLVM_CXX_FLAGS := -I/usr/lib64/llvm6.0/include -O2 -g -pipe -Wall -D__STDC_LIMIT_MACROS
LLVM_LD_FLAGS := -L/usr/lib64/llvm6.0/lib
LLVM_LIBS := -Wl,--start-group -lLLVM-10 -lLLVM-10.0.0 -lLLVM -Wl,--start-group

Clang 10 is linked against llvm-10 so I get strange errors when running crystal spec.
Also /bin/llvm-config points to the version 10 but it's not selected.

Undefined reference when building bindgen

I'm trying to build ubuntu docker and getting this error:

/usr/lib/llvm-5.0/bin/../lib/libclangEdit.a(Commit.cpp.o): In function `clang::edit::Commit::canRemoveRange(clang::CharSourceRange, clang::edit::FileOffset&, unsigned int&)':
(.text._ZN5clang4edit6Commit14canRemoveRangeENS_15CharSourceRangeERNS0_10FileOffsetERj+0xda): undefined reference to `clang::PPConditionalDirectiveRecord::rangeIntersectsConditionalDirective(clang::SourceRange) const'
/usr/lib/llvm-5.0/bin/../lib/libclangEdit.a(Commit.cpp.o): In function `clang::edit::Commit::insertFromRange(clang::SourceLocation, clang::CharSourceRange, bool, bool)':
(.text._ZN5clang4edit6Commit15insertFromRangeENS_14SourceLocationENS_15CharSourceRangeEbb+0xa7): undefined reference to `clang::PPConditionalDirectiveRecord::findConditionalDirectiveRegionLoc(clang::SourceLocation) const'
/usr/lib/llvm-5.0/bin/../lib/libclangEdit.a(Commit.cpp.o): In function `clang::edit::Commit::insertFromRange(clang::SourceLocation, clang::CharSourceRange, bool, bool)':
(.text._ZN5clang4edit6Commit15insertFromRangeENS_14SourceLocationENS_15CharSourceRangeEbb+0xb5): undefined reference to `clang::PPConditionalDirectiveRecord::findConditionalDirectiveRegionLoc(clang::SourceLocation) const'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Similar error when trying to build by hand.

Support nested anonymous types

Currently, Bindgen doesn't support nested anonymous types at all:

struct T {
  struct { int x, y; } point;
  struct { float u, v; };
};

classes: { T: T }
types:
  T: { copy_structure: true }

lib Binding
  struct T
    point : T::(anonymous)*
    unnamed_arg_0 : T::(anonymous)*
  end

  # SanityCheck will complain about the inaccessible (anonymous) types
  # x, y, u, and v are entirely missing
end

There are many examples of these types in the wild, mostly C struct / union compounds: (union support would be in a separate issue)

/**
 *  Get the SDL joystick layer binding for this controller button/axis mapping
 */
typedef struct SDL_GameControllerButtonBind
{
    SDL_GameControllerBindType bindType;
    
    union
    {
        int button;
        int axis;
        struct {
            int hat;
            int hat_mask;
        } hat;
    } value;

} SDL_GameControllerButtonBind;

Successfully parsing the nested types is the first step to supporting these classes, whether they are wrapped or not. This issue could be broken down into several tasks:

• Allow the Clang parser to emit nested anonymous types. The only chance to emit those types is while parsing the outer type, otherwise it would be quite difficult to map them to data members (point and unnamed_arg_0 end up having the same generated type name). #84 is a step towards this.
• These anonymous types obviously cannot be named inside the YAML configuration files, so Bindgen needs to generate appropriate rules for them (my guess is either copy the structure only, or derive from the parent type's rules).
• If an anonymous type does not define a data member, its structure should be copied to the parent type.
• C/C++ methods cannot reference anonymous types, but the property methods from Crystal wrappers can, so something has to be done about it.

At the end, Binding::T should look like the following:

lib Binding
  struct T
    point : T_Unnamed0
    u : Float32
    v : Float32
  end
  struct T_Unnamed0
    x : Int32
    y : Int32
  end
end

Or if there is a Crystal wrapper:

class T
  @unwrap : Binding::T*

  def point : ??? end
  def point=(point : ???) end

  def u : Int32 end
  def u=(u : Int32) : Void end
  def v : Int32 end
  def v=(v : Int32) : Void end
end

Use built in option parser for find_clang.cr

https://crystal-lang.org/api/0.34.0/OptionParser.html

require "option_parser"

upcase = false
destination = "World"

OptionParser.parse do |parser|
  parser.banner = "Usage: salute [arguments]"
  parser.on("-u", "--upcase", "Upcases the salute") { upcase = true }
  parser.on("-t NAME", "--to=NAME", "Specifies the name to salute") { |name| destination = name }
  parser.on("-h", "--help", "Show this help") do
    puts parser
    exit
  end
  parser.invalid_option do |flag|
    STDERR.puts "ERROR: #{flag} is not a valid option."
    STDERR.puts parser
    exit(1)
  end
end

destination = destination.upcase if upcase
puts "Hello #{destination}!"

Windows support

Trying to compile bindgen on Windows produces first error in find_clang.cr on:

UNAME_S = `uname -s`.chomp

fatal error: 'memory' file not found

Note: I've successfully got bindgen to work under macos. I had to modify the scripts a bit, but I was able to compile clang and everything. I just couldn't get -ltinfo, I just removed it for now.

I don't think I'll succeed, but I'm trying to generate bindings for libv8.

One of the first things it includes is <memory>. Which is part of the stdlib. My .yml config looks like this:

module: V8
library: "%/ext/binding_{BINDING_PLATFORM}.a -lstdc++ -lv8_base -lv8_init -lv8_initializers -lv8_libbase -lv8_libplatform -lv8_libsampler -lv8_nosnapshot"

processors:
  # Graph-refining processors:
  - default_constructor # Create default constructors where possible
  - function_class # Turn OOP-y C APIs into real classes
  - inheritance # Mirror inheritance hierarchy from C++
  - copy_structs # Copy structures as marked
  - macros # Support for macro mapping
  - functions # Add non-class functions
  - filter_methods # Throw out filtered methods
  - extern_c # Directly bind to pure C functions
  - instantiate_containers # Actually instantiate containers
  - enums # Add enums
  # Preliminary generation processors:
  - crystal_wrapper # Create Crystal wrappers
  - virtual_override # Allow overriding C++ virtual methods
  - cpp_wrapper # Create C++ <-> C wrappers
  - crystal_binding # Create `lib` bindings for the C wrapper
  - sanity_check # Shows issues, if any

generators:
  # C++ generator
  cpp:
    # Output file path  (Mandatory)
    output: ext/my_bindings.cpp
    # Output file preamble  (Optional)
    preamble: |-
      #include "bindgen_helper.hpp"
  crystal:
    # You'll most likely only need the `output` option.
    output: src/v8/binding.cr
  
parser:
  # List of files to include.  Can be relative to search-paths.
  # This is the only required option:
  files:
    - stdlib.h
    - v8.h
  flags: [ "-x", "c++", "-std=c++11" ]

I get the following error:

$ crystal ./lib/bindgen/src/bindgen.cr -- --chdir $(pwd) v8.yml
In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.GKCo5a:3:
/usr/local/include/v8.h:21:10: fatal error: 'memory' file not found
#include <memory>
         ^~~~~~~~

How can I make this work? I tried adding memory.h to my files and it didn't work.

[proposal] Update and automate documentation

What the title says. There is a lot of great documentation all in the one README file. perhaps we can break it up into smaller sections and use the wiki?

Issues with container spec

Seems aliasing is broken, unsure when it broke, but it should be reproducible with the container spec.

$ crystal spec spec/integration/containers_spec.cr
+ rm -f '*.o' containers.cpp containers_test.cr
Invalid alias name "std::vector<int>" at Test::Binding::std::vector<int>
Invalid alias name "std::vector<std::vector<int>>" at Test::Binding::std::vector<std::vector<int>>
Invalid alias name "std::vector<std::string>" at Test::Binding::std::vector<std::string>
Invalid alias name "std::vector<rgb>" at Test::Binding::std::vector<rgb>
Invalid alias name "std::vector<double>" at Test::Binding::std::vector<double>
Result type std::vector<int> is unreachable at Test::Binding#integers()
Result type std::vector<std::vector<int>> is unreachable at Test::Binding#grid()
Result type std::vector<std::string> is unreachable at Test::Binding#strings()
Result type std::vector<rgb> is unreachable at Test::Binding#palette()
Argument 2 has unreachable type std::vector<double> at Test::Binding#sum(list)
Result type Binding::std::vector<int> is unreachable at Test::Containers#integers()
Result type Binding::std::vector<std::vector<int>> is unreachable at Test::Containers#grid()
Result type Binding::std::vector<std::string> is unreachable at Test::Containers#strings()
Result type Binding::std::vector<rgb> is unreachable at Test::Containers#palette()
Argument 1 has unreachable type Binding::std::vector<double> at Test::Containers#sum(list)
Found 15 errors.  Aborting.
F

Failures:

  1) container instantiation feature works
     Failure/Error: tool.run!.should eq(0)

       Expected: 0
            got: 1

     # spec/integration/spec_helper.cr:80

Finished in 611.09 milliseconds
1 examples, 1 failures, 0 errors, 0 pending

Failed examples:

crystal spec spec/integration/containers_spec.cr:4 # container instantiation feature works

CrystalProc ignores Crystal-side type conversions

Currently the converter, from_crystal, and to_crystal configurations only apply to direct uses of binding types, not to Proc types accepting or returning these types, including block arguments. This means all Qt signal handlers involving QString will fail, for example:

require "qt5"

qApp = Qt::Application.new
wnd = Qt::Widget.new
wnd.on_window_title_changed do |str|
  puts str
end
wnd.window_title = "string arg test"
wnd.show
Qt::Application.exec
# prints gibberish and most likely crashes

as the type conversion applied by Qt::Converter::QString is lost through CrystalProc:

module Qt
  class Widget
    def on_window_title_changed(&_proc_ : Proc(String, Void)) : SignalConnection
      # 1. _proc_ expects a Crystal String
      SignalConnection.new(unwrap: Binding.bg_QWidget_CONNECT_windowTitleChanged_CrystalProc_void_const_QString_R(self, BindgenHelper.wrap_proc(_proc_)))
    end
  end

  lib Binding
    # 2. _proc_ expects a Crystal String
    fun bg_QWidget_CONNECT_windowTitleChanged_CrystalProc_void_const_QString_R(_self_ : QWidget*, _proc_ : CrystalProc) : QMetaObjectConnection*
  end
end

extern "C" QMetaObject::Connection * bg_QWidget_CONNECT_windowTitleChanged_CrystalProc_void_const_QString_R(QWidget * _self_, CrystalProc<void, const CrystalString> _proc_) {
  // 3. qstring_to_crystal produces Qt::Binding::CrystalString, not String
  //    (this conversion is due to from_cpp/to_cpp)
  return new (UseGC) QMetaObject::Connection (QObject::connect(_self_, (void(QWidget::*)(const QString &))&QWidget::windowTitleChanged, [_proc_](const QString & title){ _proc_(qstring_to_crystal(title)); }));
}

I was able to get the correct behaviour by manually patching the converter in:

module Qt
  class Widget
    def on_window_title_changed(&_proc_ : Proc(String, Void)) : SignalConnection
      SignalConnection.new(unwrap: Binding.bg_QWidget_CONNECT_windowTitleChanged_CrystalProc_void_const_QString_R(self, BindgenHelper.wrap_proc(->(title : Qt::Binding::CrystalString) do
        _proc_.call(Qt::Converter::QString.unwrap(title))
      end)))
    end
  end
end

This is conceptually similar to the lambda expression that appears on the C++ side, which is also Qt-specific behaviour, through Bindgen::CallBuilder::CppQobjectConnect. In general every CrystalProc needs conversion on both sides, {from,to}_cpp between binding types and C++, and {from,to}_crystal | converter between binding types and Crystal, not just the ones the Qt processor uses (it doesn't seem non-Qt bindings can produce CrystalProcs though).

find_clang.cr, function parse_clang_output takes only first include path while skipping others

For example I have this flags (https://github.com/Papierkorb/bindgen/blob/master/clang/find_clang.cr#L380), I'll include only -internal-isystem part:

...
 "-internal-isystem",
 "/usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10",
 "/usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/x86_64-redhat-linux",
 "/usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/backward",
 "/usr/local/include",
 "/usr/lib64/clang/10.0.0/include",
 "-internal-externc-isystem",
...

And this function returns only first one dir - /usr/include/c++/10, skipping the /usr/local/include, /usr/lib64/clang/10.0.0/include and others.
And because I don't have /usr/lib64/clang/10.0.0/include in include paths defined, later I see this error running crystal spec:

In file included from /tmp/.kMdBOGbindgen:1:
In file included from /home/zawertun/tmp/bindgen@Papierkorb/spec/integration/c_wrapper.cpp:1:
In file included from /usr/lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/cstdlib:75:
/usr/include/stdlib.h:31:10: fatal error: 'stddef.h' file not found
#include <stddef.h>
         ^~~~~~~~~~

File stddef.h can be found in the /usr/lib64/clang/10.0.0/include dir, if I add this include manually - crystal spec finishes without errors.

Update or remove the Contributors section in the README

With many people lately doing a ton of work on this project, y'all are pretty humble when it comes to adding yourself to the list. Please do so! Or remove the section, I leave that up the community.

Cheers!

Support downcasting between Crystal wrapper types

Previously we discussed the possibility of recovering Crystal wrapper instances/types from their @unwrap pointers, but it turns out that only solves part of the marshalling problem; if there wasn't a wrapper instance in the first place, we are left with a C pointer in Crystal with no access to C++'s RTTI. This issue attempts to address this other part. Consider:

struct A {
  virtual ~A() { }
  A *create();
};

struct B : A { };
struct C : A { };

A *A::create() {
  return new B;
}

module Test
  class A
    def create : A
      A.new(unwrap: Binding.bg_A_create_(self))
    end
  end

  class B < A end
  class C < A end
end

x = Test::A.new.create
x.as?(Test::B) # returns `nil`, runtime type of `x` is `Test::A` (or even `Test::AImpl`)
x.unsafe_as(Test::C) # bad idea

To that end I suggest wrapping dynamic_cast for every possible downcast from one wrapped polymorphic type to another. For example, the cast from A to B would look like:

extern "C" B * bg_A__CAST_B_(A * _self_) {
  return dynamic_cast<B *>(_self_);
}

module Test
  lib Binding
    alias A = Void
    alias B = Void
    fun bg_A__CAST_B_(_self_ : A*) : B*
  end

  class A
    def to?(_type_ : B.class) : B?
      ptr = Binding.bg_A__CAST_B_(self)
      B.new(unwrap: ptr) unless ptr.null?
    end
  end

  class B < A
    # cast is not needed from here, because `Test::B` and
    # its subclasses always wrap a `B` instance from C++
    # the return type is also no longer nilable
    def to?(_type_ : B.class) : B
      self
    end
  end
end

x.to?(Test::B) # => #<Test::B:...>
x.to?(Test::C) # => nil

Taking inspiration from block overloads, these cast methods take the target wrapper class itself as an argument. The C++ wrappers always perform casts using pointers; bad casts can be reported on the Crystal side with #not_nil!.

For a linear hierarchy of n classes (Tn < Tn-1 < ... < T2 < T1), a naive approach would generate a total of O(n²) #to? methods. This can be reduced to O(n) by noting that pointers to base types are also pointers to derived types, so that e.g. T1#to?(T4.class) can be reused in T2 and T3 and therefore does not have to be redefined in those subclasses. (The case with multiple inheritance will be more complicated.)

No special handling is needed for abstract wrappers; the Impl classes will pick up the #to? methods from the abstract classes they inherit from. Code like this will finally be possible:

def all_groups(scene : Qt::GraphicsScene)
  scene.items.compact_map &.to?(Qt::GraphicsItemGroup)
end

As part of the changes I'd suggest that the existing upcast methods for classes with multiple bases (the #as_X methods generated by the Inheritance processor) also take the #to? form. Note that they already share a similar C++ body, using static_cast instead of dynamic_cast.

Better CMake use of find_clang.cr

Currently, CMake calls find_clang.cr 3 times and also parses its stdout.
This causes the cmake run time to be longer than needed, and also it makes find_clang.cr unable to print meaningful debug/info messages because its output is parsed directly.

We can improve this in the following way:

1. Make CMake call find_clang.cr only once
2. Make find_clang.cr generate all of its content at once and dump it into various files on disk. (Currently it already generates 3 files, but it could generate at least 2 more - one with one for output of --print-clang-libs and one for output of --print-llvm-libs)
3. CMake should then read in those file
4. find_clang.cr will then be able to print info/notices to stdout, such as the value of its autodiscovery of BINDGEN_DYNAMIC setting as well as the paths/filenames of all the files it has generated

Issues wrapping namespaced classes

I'm having some issues trying to create bindings for the PoDoFo library - everything is name-spaced under PoDoFo. I'm mainly having issues with classes.

If I use the namespace in the config, I have two issues (that I've found):

• Inheritance is not detected in the crystal wrapper
• the C++ bindings using declaration uses the namespace improperly

podofo.yml

module: Podofo # I also have issues if this is the same as the actual namespace

classes:
  PoDoFo::PdfDocument: Document
  PoDoFo::PdfMemDocument: MemDocument
  PoDoFo::PdfStreamedDocument: StreamedDocument

src/podofo/binding.cr

module Podofo
  ...
  class Document
    ...
  end

  class MemDocument
    ...
  end

  class StreamedDocument
    ...
  end
end

ext/podofo_binding.cpp

struct BgInherit_PdfMemDocument : public PoDoFo::PdfMemDocument {
  using PoDoFo::PdfMemDocument::PoDoFo::PdfMemDocument;

If I manually edit the generated code to fix these issues I can run crystal code using the bindings.

If I don't use the namespace, the inheritance is detected but the cpp bindings are incorrect since the namespace isn't used

podofo.yml

module: Podofo

classes:
  PdfDocument: Document
  PdfMemDocument: MemDocument
  PdfStreamedDocument: StreamedDocument

src/podofo/binding.cr

module Podofo
  ...
  class Document
    ...
  end

  class MemDocument < Document
    ...
  end

  class StreamedDocument < Document
    ...
  end
end

ext/podofo_binding.cpp

struct BgInherit_PdfMemDocument : PdfMemDocument {
  using PdfMemDocument::PdfMemDocument;

I've also tried messing around by setting the types manually for each class, but didn't seem to make any difference.

types:
  PdfDocument:
    cpp_type: PoDoFo::PdfDocument
    crystal_type: Document
  # etc

From what I can tell, qt doesn't namespace its classes, so I'm wondering if this is something that was never really implemented since it wasn't needed. I'd be willing to help out, I don't have a ton of experience with code parsing/generation so I'd need a little direction. I also don't have a ton of c++ experience (which is why I'm trying to create crystal bindings 😄 ), I'm mostly a ruby developer but we have some c++ that I have to maintain from time to time.

Also, I have to turn off the sanity_check processor or else everything blows up about invalid aliases and unreachable types for the specified classes.

Issue with newer Clangs.

I'm trying to get this working with a newer clang (My system has clang 7 or 8, cant remember , I'm on a different machine right now.

First issue is

https://www.mail-archive.com/[email protected]/msg73202.html

QualTypeNames.h has been moved into AST

Also in include/clang_type_name.hpp theres a reference to and the compiler isnt sure if the right namespace refers to clang::PointerType or llvm::PointerType

out of unfamiliarity I just replaced it with clang::PointerType and it compiled. But I dont actually know if it'll explode when pressed or not. I'm just not particularly familiar with C++, alas.

I'd push a pull request but I'm not currently sure how :/

Header in files autogenerated by find_clang.cr

In files find_clang.cr autogenerates, header should be verified to exist which explains that the file was autogenerated and by which file/script.

Bindgen catches segmentation fault

When executing tool.sh, I get the following error:

Unhandled exception: clang/bindgen failed to execute. (Exception)
  from src/bindgen/parser/runner.cr:38:11 in 'run'
  from src/bindgen/parser/runner.cr:45:9 in 'run_and_parse'
  from src/bindgen/tool.cr:130:7 in 'parse_cpp_sources'
  from src/bindgen/tool.cr:48:7 in 'run_steps'
  from src/bindgen/tool.cr:40:15 in 'run!'
  from src/bindgen.cr:58:1 in '__crystal_main'
  from /usr/lib/crystal/crystal/main.cr:97:5 in 'main_user_code'
  from /usr/lib/crystal/crystal/main.cr:86:7 in 'main'
  from /usr/lib/crystal/crystal/main.cr:106:3 in 'main'
  from __libc_start_main
  from _start
  from ???

Going deeper with LLDB:
lldb lib/bindgen/clang/bindgen -- ..args..
Gives the following result:

Process 30900 stopped
* thread #1, name = 'bindgen', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x00007ffff60a2599 libclangLex.so.9`clang::Preprocessor::RemovePragmaHandler(llvm::StringRef, clang::PragmaHandler*) + 137
libclangLex.so.9`clang::Preprocessor::RemovePragmaHandler:
->  0x7ffff60a2599 <+137>: movq   (%rax), %rax
    0x7ffff60a259c <+140>: callq  *0x18(%rax)
    0x7ffff60a259f <+143>: movq   %rax, %rbp
    0x7ffff60a25a2 <+146>: jmp    0x7ffff60a2526            ; <+22>

Support for Other Languages?

Nice architecture! Shows what can be done.

Given everything is modular, is there much chance of supporting other languages? Don't know if it would be a lot of work, but an (almost) language-agnostic framework would be really nice. I am thinking Rust would be an obvious one, perhaps to wrap a whole crate.

Generated `void *` function not returning anything

When generating qt5.cr I get the following code generated:

extern "C" void * bg_QWidget_qt_metacast_const_char_X(QWidget * _self_, const char * unnamed_arg_0) {
  _self_->qt_metacast(unnamed_arg_0);
}

Meanwhile, the pre-generated branch has the following:

extern "C" void* bg_QWidget_qt_metacast_const_char_X(QWidget *_self_, const char* unnamed_arg_0) {
  return _self_->qt_metacast(unnamed_arg_0);
}

$ llvm-config --version
5.0.0
$ clang --version
clang version 5.0.0 (tags/RELEASE_500/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
$ crystal --version
Crystal 0.23.1 (2017-09-10) LLVM 5.0.0

bindgen$ git status
HEAD detached at 8c946eb
nothing to commit, working tree clean

crystal-qt$ git status
HEAD detached at a570e69
nothing to commit, working tree clean

Code used to generate:

(cd ../crystal-qt && ../bindgen/tool.sh qt.yml && egrep -A2 "bg_QWidget_qt_metacast_const_char_X\(Q" ext/qt_binding.cpp)

Runtime error after linking to .a and .so

Hey @kalinon, on LLVM 7, when I compile bindgen, it cannot be started due to:

./bindgen 
: CommandLine Error: Option 'help-list' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

This happens if either it links against different/multiple versions of LLVM, or if it links both .a and .so libs of the same version. In my case it's the second issue.

I've ran LDFLAGS=-v cmake .; make -j which showed me the linker line. It is pasted here:
https://bpa.st/IJNQ

In it, it's clear that it links against a bunch of .a files, but then also against libLTO.so and libLLVM-7.so. The llvm.so is not linked to only when I remove both llvm.so and liblto.so from linked flags. (I am not sure if lto brings in llvm.so as a dependency or both of them get incorrectly linked to.)

I ran clang tool in debug mode (crystal find_clang.cr -- --debug) and it shows LLVMLTO as a library on the list, but I don't think this is responsible for the .so objects, because it also links against lto.a (i.e. I think the detected libraries are correct, and the erroneous .so links must be coming from somewhere else).

Would you have an idea where the .so things are coming from?

Currently I'm trying to determine this myself by looking at the position of it in the link list, to figure out which variable it came from.

Stack overflow due to calling superclass method inside virtual override

I was doing work that required an overriding method to call the overridden method in the superclass (Qt::Object::eventFilter from qt5.cr to be exact), but to no avail. Suppose this class is in spec/integration/virtual_override_spec.cr:

class OverrideThing < Test::Base
  def calc(a, b) # (1)
    super * (a - b)
  end
end

# ...

OverrideThing.new.calc(10, 4).should eq(84) # (0)

This leads to a stack overflow:

...
from virtual_override_spec.cr:30:11 in 'calc'     # (1)
from tmp/virtual_override.cr:145:5 in '->'        # (6)
from _ZNK11CrystalProcIiJiiEEclEii                # (5)
from _ZN14BgInherit_Base4calcEii                  # (4)
from bg_Base_calc_int_int                         # (3)
from tmp/virtual_override.cr:131:7 in 'calc'      # (2)
from virtual_override_spec.cr:30:11 in 'calc'     # (1)
from tmp/virtual_override.cr:145:5 in '->'        # (6)
from _ZNK11CrystalProcIiJiiEEclEii                # (5)
from _ZN14BgInherit_Base4calcEii                  # (4)
from bg_Base_calc_int_int                         # (3)
from tmp/virtual_override.cr:131:7 in 'calc'      # (2)
from virtual_override_spec.cr:30:11 in 'calc'     # (1)
from virtual_override_spec.cr:78:11 in '->'       # (0)

The bodies of the relevant generated methods are:

module Test
  @[Link(ldflags: "#{__DIR__}/../tmp/virtual_override.o -lstdc++")]
  lib Binding
    fun bg_Base_calc_int_int(_self_ : Base*, a : Int32, b : Int32) : Int32 # (3)
  end

  class Base
    def initialize()
      # ...
      jump_table = Binding::BgJumptable_Base.new(
        bg_Base_calc_int_int: BindgenHelper.wrap_proc(
          Proc(Int32, Int32, Int32).new{|a, b| self.calc(a, b) } # (6)
        ),
        # ...
      )
      Binding.bg_BgInherit_Base_JUMPTABLE_BgJumptable_Base_R(result, pointerof(jump_table))
    end

    def calc(a : Int32, b : Int32) : Int32 # (2)
      Binding.bg_Base_calc_int_int(self, a, b)
    end
  end
end

struct BgInherit_Base : public Base {
  using Base::Base;
  BgJumptable_Base bgJump;
  int calc(int a, int b) override { // (4)
    BgInherit_Base *_self_ = this;
    if (_self_->bgJump.bg_Base_calc_int_int.isValid()) {
      return _self_->bgJump.bg_Base_calc_int_int(a, b);
    } else {
      return Base::calc(a, b);
    }
  }
};

extern "C" int bg_Base_calc_int_int(Base * _self_, int a, int b) { // (3)
  return _self_->calc(a, b);
}

template<typename T, typename ... Args>
struct CrystalProc {
  T operator()(Args ... arguments) const { // (5)
    if (this->self) {
      return this->withSelf(this->self, arguments...);
    } else {
      return this->withoutSelf(arguments...);
    }
  }
};

If the method is simply overridden, calls should stop at (1) immediately; if an override isn't provided, the jump table entry for calc would be invalid, so C++'s Base::calc would be invoked after (4). It looks like BgInherit_Base::calc always invokes the most derived calc method, so it cannot distinguish between super and a regular call.

'crystal deps' no longer exists

When I add bindgen to a project and run 'shards', the following happens:

Fetching https://github.com/Papierkorb/bindgen.git
Fetching https://github.com/Papierkorb/toka.git
Installing bindgen (0.6.1)
Postinstall crystal deps
Failed crystal deps:
Please use 'shards': 'crystal deps' has been removed

This is on Crystal 0.26.0 with a new/empty project created with 'crystal init app'.
This message happens only with bindgen, not with other shards.

Any ideas? Thanks!

[Question] Dealing with classes containing template arguments

Hi, sorry, If this is a noob question. I'm trying to bind to a library, which uses chromes v8 base. The class, which is causing me problems is: scoped_refptr can be seen here. It's not a valid camel case name, so when something like scoped_refptr<nu::MenuBar> is seen, it can't convert it.

I tried adding it to classes property, but I then get:

/tmp/.bk5Cpxbindgen:43:32: error: use of class template 'scoped_refptr' requires template arguments
template class BindgenTypeInfo<scoped_refptr>;
                               ^
/home/augustinas/personal/native-ui/ext/yue/include/base/memory/scoped_refptr.h:175:7: note: template is declared here
class scoped_refptr {
      ^
Segmentation fault (core dumped)

What is the correct way to add a definition for it?.

/usr/bin/ld: missing --end-group; added as last command line option

Warning on Debian 9 and LLVM 7:

[..%] Linking CXX executable bindgen
/usr/bin/ld: missing --end-group; added as last command line option
/usr/bin/ld: missing --end-group; added as last command line option
[100%] Built target bindgen

Potential premature collection of wrapped types

Consider these snippets:

struct Inner {
  Inner(int x) : x(x) { }

  int x;
  // other members
};

struct Outer {
  static Outer *new_no_gc() { return new Outer; }

  Outer() { last_ = this; }

  Inner *inside = nullptr;
  // other members

  static Outer *last_;
  static Outer *last() { return last_; }
};

Outer *Outer::last_ = nullptr;

Test::Outer.new_no_gc.inside = Test::Inner.new(7)
GC.collect
puts Test::Outer.last.inside.x # => ???

Boehm GC's readme states:

Any objects not intended to be collected must be pointed to either from other such accessible objects, or from the registers, stack, data, or statically allocated bss segments.

The Test::Outer and Test::Inner Crystal wrappers are always GC-enabled, so they should be collected as no Crystal variables point to them. This leaves only the actual Outer and Inner instances on the C++ side. Now the Inner instance:

• is GC-enabled (because the generated _CONSTRUCT function invokes the UseGC allocator);
• is pointed to from Outer::last_, but this doesn't make it accessible because Outer::last_ itself is unmanaged (the C++ wrapper for Outer::new_no_gc simply forwards the returned pointer);
• is no longer pointed to from the managed Test::Inner instance.

Thus, it too will be collected despite being indirectly traceable through Outer::last_->inside. The last line therefore refers to a potentially deleted object, and I managed to get this snippet to crash by enlarging the Inner struct and repeating the collection step multiple times.

In this particular case, we could simply add UseGC to all occurrences of the default new operator, or use GC_malloc_uncollectable for Outer instead (the instance itself will be unmanaged, but its inside member is considered by the GC). This is clearly not possible if only the library and header files are available to Bindgen.

Qt5.cr is also prone to this issue. Example:

QStatusBar *QMainWindow::statusBar() const
{
    QStatusBar *statusbar = d_func()->layout->statusBar();
    if (!statusbar) {
        QMainWindow *self = const_cast<QMainWindow *>(this);
        statusbar = new QStatusBar(self); // unmanaged
        statusbar->setSizePolicy(QSizePolicy::Ignored, QSizePolicy::Fixed);
        self->setStatusBar(statusbar);
    }
    return statusbar;
}

window : Qt::MainWindow
window.status_bar.add_widget(Qt::Label.new "123")
# the constructed QLabel will be collected!

Crash in clang destructor

@@ -100,6 +100,9 @@ void BindgenASTConsumer::serializeAndOutput() {

        stream << "macros" << JsonStream::Separator << this->m_macros;  // "macros": [ ... ]
        stream << JsonStream::ObjectEnd; // }
+
+       // FIXME: Currently the process crashes during clang's Parser destructor. This is a workaround.
+       exit(0);
 }

Renewed maintenance of bindgen

Hello @acoolstraw, @jreidinger , @kalinon , @lbguilherme , @ZaWertun

Stefan was kind to give me write access to the bindgen/qt5 repository. I would like to work on bindgen so that it builds again and runs tests successfully, and then to produce current/working Qt bindings.

Any interest or contributions from you would be much appreciated - either in the form of PRs or general comments/insights (possibly added to this thread or whatever is more appropriate).

Let's try to build a critical mass here so that we succeed in updating the Qt bindings.

(For example, reviewing your existing PRs or updating your forks to the latest version of bindgen so that it is easier to identify your own changes/improvements would be a great start, and we could coordinate from there.)

Also let's please keep any patches/PRs small and isolated so that they are easier to review and agree on.

Thanks!

Destructors not called on GC-enabled instances

At no point during execution does Bindgen actually invoke the destructors of wrapped C++ instances; UseGC only ensures the instances use the GC heap, deletion is still performed by a raw GC_free. Consider:

#include <iostream>
#include <gc/gc_cpp.h>

struct T {
  virtual ~T() { std::cout << "T::~T()\n"; }
};

extern "C" T * bg_T__CONSTRUCT_() {
  return new (UseGC) T();
}

lib Binding
  alias T = Void
  fun bg_T__CONSTRUCT_() : T*
end

x = Binding.bg_T__CONSTRUCT_()
x = nil
5.times {GC.collect} # a plain collection won't work, perhaps because the pointer would then be on the stack

Nothing happens, but if a clean-up function is supplied to the constructor:

extern "C" void bg_T__DESTRUCT_(void * _self_, void * client_data) {
  reinterpret_cast<T *>(_self_)->~T();
}

extern "C" T *bg_T__CONSTRUCT_() {
  return new (UseGC, bg_T__DESTRUCT_) T();
}

Then T::~T() will be printed. Is this behaviour intentional? (To be fair, since Qt has its own ownership semantics, calling the destructors on returned wrappers would probably break it immediately.)

9223372036854775808 in a macro is expected to evaluate to 9223372036854775808_u64 but gets interpreted as an Int64 instead

There is still one issue that remains, when running the specs:

Failures:

  1) clang tool macros feature exports the macros

       At macros.10.evaluated: Expected 9223372036854775808_u64, got -9223372036854775808_i64 (ClangValidationError)
         from /spec_helper.cr:79:91 in 'check_partial_value'
         from /spec_helper.cr:64:7 in 'check_partial_value'
         from /spec_helper.cr:75:7 in 'check_partial_value'
         from spec/clang/spec_helper.cr:31:5 in 'clang_tool:macros'
         from spec/clang/macros_spec.cr:5:5 in '->'
         from /usr/share/crystal/src/spec/methods.cr:255:3 in 'it'
         from spec/clang/macros_spec.cr:4:3 in '->'
         from /usr/share/crystal/src/spec/context.cr:255:3 in 'describe'
         from /usr/share/crystal/src/spec/methods.cr:16:5 in 'describe'
         from spec/integration/spec_helper.cr:1:1 in '__crystal_main'
         from /usr/share/crystal/src/crystal/main.cr:97:5 in 'main_user_code'
         from /usr/share/crystal/src/crystal/main.cr:86:7 in 'main'
         from /usr/share/crystal/src/crystal/main.cr:106:3 in 'main'
         from __libc_start_main
         from _start
         from ???

I'm really not sure why that's happening but it could cause some nasty bugs

Does not work with LLVM/Clang 14 and multi-LLVM setup

There are some changes in LLVM/Clang 14 that need CPP adjustments for parser. Also, the LLVM libs finder does not work properly if multiple LLVM runtime libraries are installed (it tries to link all of them).

Overloaded Qt signals

While using qt5.cr I noticed that #on_SIGNAL methods do not work properly if the signal has multiple signatures. One example is the overloads of QComboBox::activated:

• void QComboBox::activated(int index)
• void QComboBox::activated(const QString &text)

The Qt processor translates them to something along the following lines:

module Qt
  lib Binding
    fun bg_QComboBox_CONNECT_activated_CrystalProc_void_int(_self_ : QComboBox*, _proc_ : CrystalProc) : QMetaObjectConnection*
    fun bg_QComboBox_CONNECT_activated_CrystalProc_void_const_QString_R(_self_ : QComboBox*, _proc_ : CrystalProc) : QMetaObjectConnection*
  end

  class ComboBox
    def on_activated(&_proc_ : Proc(Int32, Void)) : SignalConnection
      SignalConnection.new(unwrap: Binding.bg_QComboBox_CONNECT_activated_CrystalProc_void_int(self, BindgenHelper.wrap_proc(_proc_)))
    end
    
    def on_activated(&_proc_ : Proc(String, Void)) : SignalConnection
      SignalConnection.new(unwrap: Binding.bg_QComboBox_CONNECT_activated_CrystalProc_void_const_QString_R(self, BindgenHelper.wrap_proc(_proc_)))
    end
  end
end

However, Crystal does not support method overloading through different type restrictions on a block, so the second on_activated definition overwrites the first, which means code like this fails to compile:

cb = Qt::ComboBox.new
cb.on_activated(&->(x : Int32) { puts "item index #{x}" })
# Error: expected block argument's argument #1 to be String, not Int32

The int overload itself works if the class is reopened and the method definition is repeated under a different name, but this is most certainly not end users are expected to do themselves.

Bindgen::Graph::Path edge cases

I found a bunch of edge cases in Bindgen::Graph::Path while trying to figure out some namespace-related issues:

describe Bindgen::Graph::Path do
  describe "#lookup" do
    context "given a global path" do
      it "starts lookup from the root" do
        path("::Root::Right::D").lookup(a).should be(d)
        path("::Root::Root::Right::D").lookup(a).should_not be(d) # fails
      end
    end
  end
end

Global path lookup is wrong if the root name is repeated twice.

#              E
#             /
#   f1  -->  F
#           / \
# f2  -->  F   G
#         /     \
#        H       J
e = Bindgen::Graph::Namespace.new("E")
f1 = Bindgen::Graph::Namespace.new("F", e)
f2 = Bindgen::Graph::Namespace.new("F", f1)
g = Bindgen::Graph::Namespace.new("G", f1)
h = Bindgen::Graph::Namespace.new("H", f2)
j = Bindgen::Graph::Namespace.new("J", h)

# these all fail
path("F::G").lookup(f1).should be_nil
path("F::G").lookup(f2).should be_nil
path("F::G").lookup(g).should be_nil
path("F::G").lookup(h).should be_nil
path("F::G").lookup(j).should be_nil

Local path lookup doesn't handle shadowed namespaces properly. (Both Crystal and C++ do this.)

Bindgen::Graph::Path.from("::").last_part # ""

This is really confusing; no places outside Path use the empty string to refer to a top-level / root namespace.

I'll soon submit a refactor that fixes these edge cases.

Undefined reference to GC_throw_bad_alloc()

Today I started receiving this error in Bindgen after performing a system update which bumps my libgc's version to 8.0.4:

/usr/bin/ld: ... /spec/integration/tmp/../tmp/instance_properties.o: in function `operator new(unsigned long, GCPlacement, void (*)(void*, void*), void*)':
instance_properties.cpp:(.text._Znwm11GCPlacementPFvPvS0_ES0_[_Znwm11GCPlacementPFvPvS0_ES0_]+0xc6): undefined reference to `GC_throw_bad_alloc()'
collect2: error: ld returned 1 exit status

This is due to ivmai/bdwgc#268 (comment). According to that issue there are 3 solutions:

• Add -lgccpp to everywhere in Bindgen that has -lgc;
• Add -lgctba to everywhere in Bindgen that has -lgc;
• Add #define GC_NEW_ABORTS_ON_OOM prior to #include <gc/gc_cpp.h>.

Bindgen rejects clang 10 because it compares versions as strings

See this commit in my branch: ZaWertun@9f51a21

OptionParser for find_clang.cr

Possibly add real option parsing to find_clang.cr. (This would be a further extension of an improvement done in 151c707 )