jacob-carlborg/dstep

A tool for converting C and Objective-C headers to D modules

DStep

DStep is a tool for automatically generating D bindings for C and Objective-C libraries. This is implemented by processing C or Objective-C header files and output D modules. DStep uses the Clang compiler as a library (libclang) to process the header files.

Download

For the latest release see: releases/latest.

Pre-compiled binaries are available for macOS and Linux as 64 bit binaries and Windows as 32 and 64 bit binaries. The Linux binaries are completely statically linked and should work on all distros. The macOS binaries are statically linked against libclang requires no other dependencies than the system libraries. They should work on macOS Mavericks (10.9) and later. The Windows binaries require to install libclang. See the releases section.

Alternatively install via Dub

License

The source code is available under the Boost Software License 1.0

Building

Posix

Requirements

• libclang - http://clang.llvm.org - 10.0.0 or 9.0.0. The idea is that the two most recent versions of libclang are supported.
• A D compiler - The latest version of DMD or LDC
• Dub http://code.dlang.org/download (also shipped with the compilers)

Building

1. Install all requirements, see above

2. Clone the repository by running:

    $ git clone https://github.com/jacob-carlborg/dstep
   

3. Run dub build

A configuration script will try to automatically locate libclang by looking through a couple of default search paths. If libclang is not found in any of the default paths, please manually invoke the configuration script and specify the path to where libclang is installed using the --llvm-path flag.

$ ./configure --llvm-path /usr/lib/llvm-4.0

Windows

Requirements

• LLVM - http://llvm.org/releases/download.html - pre-built binaries for Windows. Has to be installed at the default location
• DMD - http://dlang.org/download.html - 2.071.0 or later
• Dub - http://code.dlang.org/download
• Visual Studio - for example Visual Studio Community

Building

1. Install all requirements, see above

2. Clone the repository by running:

    $ git clone git://github.com/jacob-carlborg/dstep.git
   

3. Run dub build --arch=x86_mscoff --build=release to build 32-bit version

4. Run dub build --arch=x86_64 --build=release to build 64-bit version

Remarks

Building 32-bit version requires a 32-bit variant of the Visual Studio toolchain to be present in PATH. The same for 64-bit. Remember to specify --arch=x86_mscoff when building 32-bit version. The architecture specification is mandatory as with the default architecture or --arch=x86 dub will try to use unsupported OPTLINK linker. OPTLINK linker requires unsupported version of libclang binaries. Remember to install LLVM to its default installation path and to add its binaries to the PATH environmental variable (otherwise you may need to change dub.json). When the program compiles under Windows but crashes at start, make sure an appropriate version of libclang.dll is available for DStep (you can validate it easily by copying dll to the directory with DStep). Here you can find more information on the topic.

Usage

$ dstep Foo.h -o Foo.d

For translating Objective-C headers add the -ObjC flag.

$ dstep Foo.h -o Foo.d -ObjC

For translating multiple files at once, simply pass all the files to dstep. In this case though, -o (if given) would point to output directory name. The directory will be created if it doesn't exist.

$ dstep Foo1.h Foo2.h Foo3.h .... FooN.h -o ./outputDirectory/

Use -h for usage information. Any flags recognized by Clang can be used.

Limitations/Known issues

• Supports translating some of the preprocessor, like: #define for simple constants, function like macros and the token concatenation operator (##)

• Doesn't translate #include to import. Imports for a few standard C headers are added automatically

• Doesn't translate C++ at all

• Umbrella headers. Some headers just serve to include other headers. If these other headers contain some form of protection, like #error, to be included directly this can cause problems for DStep

• Some headers are designed to always be included together with other header files. These headers may very well use symbols from other header files without including them itself. Since DStep is designed to convert header files one-by-one this doesn't work. There are two workarounds for this:

  1. Add #include-directives for the header files the header file is actually using
  2. Use the -include <file> flag available in Clang to indicate the given <file> should be processed before the file that should be translated. DStep accepts all flags Clang accepts

dstep's Issues

Typedefs and function pointers

This is an excerpt from a header from TIBCO library (here tibrv_u32 is an uint, and tibrvMsgField is a struct):

typedef tibrv_status                                // line 36 
(*tibrvMsgData_Encoder)(                            // line 37
    char**                      wire_buffer,
    tibrv_u32                   mem_available,
    tibrvMsgField*              field);

which results in

> dstep handler.h
File(893438, "")handler.h:37:3: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
File(893438, "")handler.h:36:9: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
File(893438, "")handler.h:37:1: error: function cannot return function type 'int (char **, tibrv_u32, tibrvMsgField *)'

Any ideas about this one?

tango

Hello. I want to install it, but get its error.

>./build.sh 
dstack/dstack/application/Application.d(9): Error: module Stdout is in file 'tango/io/Stdout.d' which cannot be read
import path[0] = dstack/mambo
import path[1] = dstack
import path[2] = dstep/driver
import path[3] = /usr/include/dmd/phobos
import path[4] = /usr/include/dmd/druntime/import
Failed: 'dmd' '-debug' '-gc' '-Idstack/mambo' '-Idstack' '-L-L.' '-L-lclang' '-L-ltango' '-L-rpath' '-L.' '-v' '-o-' 'dstep/driver/DStep.d' '-Idstep/driver

Treatment of #define

From what I can see #define is removed but the values of them are used. So the C code:

define LENGTH 4

char a[LENGTH];

becomes the D code:

char[4] a;

Might it be better to be more careful with these values and create an immutable variable to replace the #define value as would be done in C++.

immutable LENGTH = 4;
char[LENGTH] a;

New multiline translation

I noticed that dstep now outputs functions multi-line. I do some post processing of the dstep outputs and the new multi-line outputs would make it difficult for me to post-process the the functions outputs in the way I usually do.

Escape D keywords for function parameters

Structs with a terminating flexible array member trigger a segfault

For example, this simple header file will get dstep to segfault:

struct foobar {
    int state;
    int info[];
};

Removing the brackets on the last member will get dstep to work again, some info on flexible array members.

Compile failure with DMD v2.0.68

I get an error when compiling with DMD v2.068. Just changing tango dependency to 1.0.3+2.068 seems to fix the issue.

va_list missing import

When there are va_list references in the .h file... you need to import core.stdc.stdarg in the resulting d file.

This may have moved at some point... as I get a superfluous import core.stdc.stdio;

D1 support.

In few places in dstep there is checking for D1 version of D (version(D1)). I believe they can be removed.

dstep throws exception analysing ruby.h on Arch Linux 64-bit

dstep -include /usr/include/linux/stddef.h -I/usr/include/ruby-2.1.0 -I/usr/include/ruby-2.1.0/x86_64-linux -I/usr/include/linux -I/usr/lib/clang/3.4/include/ /usr/include/ruby-2.1.0/ruby.h
dstep.core.Exceptions.DStepException@dstep/core/Exceptions.d(13): An unknown error occurred
----------------
dstep(void dstep.driver.Application.Application.run()+0x102) [0x54fa42]
dstep(int dstack.application.Application.Application.debugStart()+0x17) [0x54e50f]
dstep(int dstack.application.Application.Application._start()+0x15) [0x54e47d]
dstep(int dstack.application.Application.Application.start!(dstep.driver.Application.Application).start(immutable(char)[][])+0x41) [0x54e5f9]
dstep(_Dmain+0x4f) [0x51266f]
dstep(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll().void __lambda1()+0x18) [0x59b81c]
dstep(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2a) [0x59b776]
dstep(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll()+0x30) [0x59b7dc]
dstep(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2a) [0x59b776]
dstep(_d_run_main+0x1a3) [0x59b6f7]
dstep(main+0x25) [0x512b2d]
/usr/lib/libc.so.6(__libc_start_main+0xf5) [0x7f0364d66b05]

structs should not be aliased by default if they are typedef

below C headers are not translated correctly because the method has the same name
as the typedef.

typedef struct rd_kafka_metadata {
        int         broker_cnt;     /* Number of brokers in 'brokers' */
        struct rd_kafka_metadata_broker *brokers;  /* Brokers */

        int         topic_cnt;      /* Number of topics in 'topics' */
        struct rd_kafka_metadata_topic *topics;    /* Topics */

        int32_t     orig_broker_id; /* Broker originating this metadata */
        char       *orig_broker_name; /* Name of originating broker */
} rd_kafka_metadata_t;


rd_kafka_metadata (rd_kafka_t *rk, int all_topics,
                   rd_kafka_topic_t *only_rkt,
                   const struct rd_kafka_metadata **metadatap,
                   int timeout_ms);

dstep not converting `const T x[]` to `const (T)* x`

I am currently working with dstep version 0.2.2 and found that: const T x[] is not currently being converted to const (T)* x. It says <unimplemented>. I am currently doing a find and replace to const T *x in the original script which gives the desired output but it would be nice not to have to do that.

Thanks a lot for this very useful tool.

ICE when building on Win32

I haven't tired on Posix yet, but with 2.060:

rdmd --build-only -ofbin\dstep -Idstack -I..\Tango-D2 dstep\driver\DStep.d
Assertion failure: '!vthis->csym' on line 727 in file 'glue.c'

with 2.059 similar ICE:

Assertion failure: '!vthis->csym' on line 686 in file 'glue.c'

Have you tried building on win32 yet? I could try running dustmite on it, although I have a good feeling it would take quite a while to reduce it.

Self alias should be removed

typedef struct Foo
{
    int a;
} Foo;

Generates:

extern (C):

alias Foo Foo;

struct Foo
{
    int a;
}

const attribute is removed from parameter of a function pointer

I ran into this when translating the following:

typedef void (*AStorageManager_obbCallbackFunc)(const char* filename, const int32_t state, void* data);

DStep turned it into this, removing the const for the second parameter:

alias void function (const(char)*, int, void*) AStorageManager_obbCallbackFunc;

I haven't checked to see if dmd will actually complain about this, but it strikes me as a bug.

Undefined symbol: clang_getEnumConstantDeclValue when running dstep

I followed all the build steps and get:

./dstep: symbol lookup error: ./dstep: undefined symbol: clang_getEnumConstantDeclValue

when i run the dstep binary.

Single space inserted after function names

It gets inserted between a function name and the opening parenthesis for its parameters and even for function pointer declarations, please remove it.

Recognise and translate __attribute__((__packed__))

In GCC and LLVM, you can declare a packed struct like so.

typedef struct __attribute__((__packed__)) { /* */ } name;

This is equilvalent to the following in D.

struct name { align(1): /* */ }

I would like it if dstep would recognise the former and translate it into the latter. Otherwise, it will output structs with the wrong alignment.

dstep dumps core on a simple header

Given a file t.h, with contents:

extern const char *const sys_errlist[];

Run dstep on it and get a core dump:

$ dstep -x c -o t.d t.c
Segmentation fault (core dumped)

I built dtep from this repo on a Debian Linux 64-bit system (after building and installing the dmd suite from its repos).

struct typedef generates recursive alias

The following header:

typedef struct Foo Foo;
struct Foo { int i; };

Will cause dstep to produce:

extern (C):

alias Foo Foo;

struct Foo
{
    int i;
}

Where alias Foo Foo is a syntax error (recursive alias definition).

I was seeing this a lot while trying to create bindings for Chipmunk2D
(example)

Embedded struct not generated.

An embedded struct within another struct does not get generated.

On the mongoose.h header, mg_request_info containing a mg_header struct:

// This structure contains information about the HTTP request.
struct mg_request_info {
  const char *request_method; // "GET", "POST", etc
  const char *uri;            // URL-decoded URI
  const char *http_version;   // E.g. "1.0", "1.1"
  const char *query_string;   // URL part after '?', not including '?', or NULL
  const char *remote_user;    // Authenticated user, or NULL if no auth used
  long remote_ip;             // Client's IP address
  int remote_port;            // Client's port
  int is_ssl;                 // 1 if SSL-ed, 0 if not
  void *user_data;            // User data pointer passed to mg_start()

  int num_headers;            // Number of HTTP headers
  struct mg_header {
    const char *name;         // HTTP header name
    const char *value;        // HTTP header value
  } http_headers[64];         // Maximum 64 headers
};

What gets generated is:

struct mg_request_info
{
    const(char)* request_method;
    const(char)* uri;
    const(char)* http_version;
    const(char)* query_string;
    const(char)* remote_user;
    c_long remote_ip;
    int remote_port;
    int is_ssl;
    void* user_data;
    int num_headers;
    mg_header[64] http_headers;
}

mg_header is left undefined.

I would have expected a definition, at least outside of mg_request_info like:

struct mg_header {
    const(char) *name;
    const(char) *value;
}

Shouldn't dstep exit with status code when there is some kind of error?

Currently dstep always returns 0 exit code, even on the failure (wrong input file/wrong command arguments). I believe it would be better, if it returns something other (e.g. 1) when there is an error. I would make dstep easier to integrate with some command line tools (and make testing easier).

publish to code.dlang.org?

Do you have any plans to make your libraries and tools available on code.dlang.org?
I think in particular this tool should be there for visibility as low-effort integration with C is a major selling point for D.

Using dub would also improve the build process imo.

`dub build` does not work

Tango dependency gets compiles with warnings enabled which breaks it:

tango.io.FileScan is deprecated. Please use tango.io.FilePath.toList or tango.io.vfs.* instead.
tango.math.BigInt is deprecated. Please use std.bigint instead.
../../.dub/packages/tango-d2port/tango/math/IEEE.d(858): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<='
../../.dub/packages/tango-d2port/tango/math/IEEE.d(1719): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '<>='
../../.dub/packages/tango-d2port/tango/math/IEEE.d(1719): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '<>='
../../.dub/packages/tango-d2port/tango/math/Bracket.d(192): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<>='
../../.dub/packages/tango-d2port/tango/math/Bracket.d(208): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<>='
../../.dub/packages/tango-d2port/tango/math/Bracket.d(235): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<>='
../../.dub/packages/tango-d2port/tango/math/Bracket.d(73): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '<>='
../../.dub/packages/tango-d2port/tango/math/Bracket.d(73): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '<>='
../../.dub/packages/tango-d2port/tango/stdc/math.d(435): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!>'
../../.dub/packages/tango-d2port/tango/stdc/math.d(436): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!>'
../../.dub/packages/tango-d2port/tango/stdc/math.d(437): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!>'
../../.dub/packages/tango-d2port/tango/stdc/math.d(440): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!>='
../../.dub/packages/tango-d2port/tango/stdc/math.d(441): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!>='
../../.dub/packages/tango-d2port/tango/stdc/math.d(442): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!>='
../../.dub/packages/tango-d2port/tango/stdc/math.d(445): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<'
../../.dub/packages/tango-d2port/tango/stdc/math.d(446): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<'
../../.dub/packages/tango-d2port/tango/stdc/math.d(447): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<'
../../.dub/packages/tango-d2port/tango/stdc/math.d(450): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<='
../../.dub/packages/tango-d2port/tango/stdc/math.d(451): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<='
../../.dub/packages/tango-d2port/tango/stdc/math.d(452): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<='
../../.dub/packages/tango-d2port/tango/stdc/math.d(455): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<>'
../../.dub/packages/tango-d2port/tango/stdc/math.d(456): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<>'
../../.dub/packages/tango-d2port/tango/stdc/math.d(457): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<>'
../../.dub/packages/tango-d2port/tango/stdc/math.d(460): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<>='
../../.dub/packages/tango-d2port/tango/stdc/math.d(461): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<>='
../../.dub/packages/tango-d2port/tango/stdc/math.d(462): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<>='
../../.dub/packages/tango-d2port/tango/text/convert/Float.d(343): Warning: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<>='
FAIL ../../.dub/packages/tango-d2port/.dub/build/static-debug-linux.posix-x86_64-dmd-F08A06E3B8138B376777099159D14BDF tango staticLibrary
Error executing command build: DMD compile run failed with exit code 1

build fails with current beta dmd (Fedora 21, 64 bit)

Any thoughts ?

tango.io.FileScan is deprecated. Please use tango.io.FilePath.toList or tango.io.vfs.* instead.
tango.math.BigInt is deprecated. Please use std.bigint instead.
../../../../root/.dub/packages/tango-1.0.0_2.066/tango/text/Regex.d(3781): Error: delegate dg (ref RegExpT!char) is not callable using argument types (RegExpT!char)
../../../../root/.dub/packages/tango-1.0.0_2.066/tango/text/Regex.d(4430): Error: template instance tango.text.Regex.RegExpT!char error instantiating
FAIL ../../../../root/.dub/packages/tango-1.0.0_2.066/.dub/build/static-debug-linux.posix-x86_64-dmd_2067-B138A764702A26D0FF777ECD4971FB9F/ tango staticLibrary
Error executing command run: dmd failed with exit code 1.

"signed char" is translated incorrectly to "wchar"

It should be byte. I'm using a prebuilt dstep 0.1.0-2 from the Arch community repo.

Great work on dstep, working well for me so far. :)

spurious generation of variadic args rather than implicit void ie () argument list

converting the mathgl headers

I got:
double mgl_rnd_ (...);

but it should just be:
double mgl_rnd();

hope this is not something simple I am doing wrong. sorry, but I have not time right now to find the original header and try to replicate it.

Spits out <unimplemented> for function-typed parameter

The following:

void freeAll(void (freeFunc)(void*));

is translated to

void freeAll (<unimplemented> freeFunc);

Instead of

void freeAll (void function(void*) freeFunc);

Build procedure on README file may needs update.

This concerns my experience at compiling dstep on an Ubuntu Linux 12.04, 32 bits system.

Tango-D2 dependence

the last version as of June 2013 no longer compiles on dmd 2.060.
Dmd 2.061 is required.

Location of libclang.so

My build of dstep could not find libclang.so. I ran sudo ldconfig <libclang directory> to be able to run it.
Note that it can also be handled by setting the LD_LIBRARY_PATH to the directory where libclang is located.

Include path issues

On the first dstep run, clang stopped with an error telling me it could
not find "stddef.h".
A quick google search seemed to indicate that I needed to upgrade from llvm-3.1 to llvm-3.2 to automagically solve that problem but after trying it, that was not the case.

I finally figured out the include path of my system using the distribution's clang (from llvm-3.0 - incompatible with dstep):

  /usr/local/include
  /usr/include
  /usr/include/i386-linux-gnu/
  /home/ray/apps/llvm-3.2/include/clang/   # wherever is installed llvm/clang
  /usr/lib/gcc/i686-linux-gnu/4.6/include/
  /usr/lib/gcc/i686-linux-gnu/4.6/include-fixed/ 

So I ran dstep like this:

$ ~/dev/dee/dstep/bin/dstep mongoose.h -o mongoose_d.d -v \
      -I/usr/local/include \
      -I/usr/include \
      -I/usr/include/i386-linux-gnu/  \
      -I/home/ray/apps/llvm-3.2/include/clang \
      -I/usr/lib/gcc/i686-linux-gnu/4.6/include/ \
      -I/usr/lib/gcc/i686-linux-gnu/4.6/include-fixed/ 

Using on the IUP header

I ran dstep on a header file (within that header's directory) including another one located within the very same directory:

$ ls -1 
...
iupdef.h
iup.h
iupkey.h
...

I got an error telling me that the included file was not found:

$ dstep iup.h -o iup_d.d -v -I [...] 
File(823D0EC, "")iup.h:13:10: error: 'iupkey.h' file not found with <angled> include; use "quotes" instead
File(823D0EC, "")iup.h:14:10: error: 'iupdef.h' file not found with <angled> include; use "quotes" instead

I solved this by adding -I$(pwd) to the command line.

Building on windows

Hello.
I installed clang x86 3.7, dub and dmd from the links given in readme.
But when i issue dub build in the dstep folder i get many warnings followed by the following error:

tango.io.FileScan is deprecated. Please use tango.io.FilePath.toList or tango.io
.vfs.* instead.
tango.math.BigInt is deprecated. Please use std.bigint instead.
not yet available for windows
dmd failed with exit code 1.

Clang, dub and dmd are all in my system path.
How do I proceed? Anything I may be missing?

Crashes if fed nonexistent header

At least with the prebuilt dstep 0.1.0-2 on Arch, it crashes and spits out a backtrace if I feed it a non-existent C header, such as if I add a typo in the header's path. It'd be nice if it issued an error instead, so the user knows what he did wrong.

Out of order typedefs of structs

Out of order typedefs of structs results in duplicated structs.

The following code:

typedef struct _Foo Foo;

struct _Foo
{
    int x;
};

Results in:

extern (C):

alias _Foo Foo;

struct _Foo
{
}

struct _Foo
{
    int x;
}

Should be:

extern (C):

alias _Foo Foo;

struct _Foo
{
    int x;
}

compilation fails with dmd git master

dstep/translator/Translator.d(93) calls the .values property, which is now a free property in object.d. This clashes with mambo.core.AssociativeArray.values

jacob-carlborg/mambo#2

Option for turning off preprocessor errors (#error)

Add option to specify include directories.

An option should be available, such as gcc's "-I", that adds a directory to search in for includes.

#define (simplest cases only?)

I'm sorry if this has already been brought up but I haven't found it in issues or docs: can preprocessor directives like #define be handled? It looks like they can't. I'm not too familiar with libclang so this may well be something obvious.

Aside from dealing with compile time flags, most of the C libraries use #define to set constants, as in #define X (1), #define Y ((float)-X), which easily transforms to immutable. For a large percentage of headers, just handling constants would be sufficient to get them working.

However, C-defines can get ugly, weird and unpredictable and sure there's no way to handle all of them... though some can be parsed into immutable constants, some into functions #define f(a, b) ((a) + (b)).

Are there any plans regarding this?

Thanks.

Option for specifying before and after code.

In the TODO list there is a point

• Option for specifying before and after code

Does it mean something specific? @jacob-carlborg do you have a specific image how it should work?

Symbol selection

It would be useful to have a way to pass a list of symbols to export (or ignore) from c source.

Generated content is only "extern(C):".

I tried dstep for header conversion, but dstep doesn't work.

/path/to/dev/dstep$ bin/dstep test_files/arrays.h
extern (C):

I built dstep with 2.059(tried many 'git co') or 2.060, but results are same...
My env is Mac OS X 10.7.5 without dvm.

Lastly, I noticed README is now incorrect.

LLVM and Clang

'git submodule add -f http://llvm.org/git/clang.git tools/clang' instead of 'git submodule add http://llvm.org/git/clang.git'.
Currenlty, above command failed.

llvm/tools$ git submodule add http://llvm.org/git/clang.git
You need to run this command from the toplevel of the working tree.

And remove clang from .gitignore.

Tango-D2

Tango-D2 doesn't include osx32 directory.
So we should do make.

$ make -f build/Makefile static-lib -j4
$ cp libtango-dmd.a <path/to/dstep>/libtango.a

wchar_t should be tranlsated to core.stdc.stddef.wchar_t

Currently wchar_t translates to wchar or dchar depending on its size. While not incorrect it would be better to translate it to core.stdc.stddef.wchar_t instead. This would make the bindings more cross-platform.

Handle typedefs of empty struct

The following code:

typedef struct _Foo Foo;

Results in:

extern (C):

alias _Foo Foo;

struct _Foo
{
}

Should be:

extern (C):

alias _Foo Foo;

struct _Foo;

Regression: clang 3.5 causes struct members to be defined again

I recently ran dstep from Arch and found that the output is different. Since the dstep package hasn't been updated, I'm guessing the update from clang 3.4 to 3.5 caused this regression. Here's a test header file I came up with to demonstrate the problem:

struct foo { int x; int y;};

struct bar { struct foo a; struct foo b;};

And the D output dstep produces:

extern (C):

struct foo
{
        int x;
        int y;
}

struct bar
{
        struct foo
        {
                int x;
                int y;
        }
        foo a;
        struct foo
        {
                int x;
                int y;
        }
        foo b;
}

There's no reason to define the struct again twice, not sure why it's doing that.

Generating code that will not compile

This is a one off case but…

The C code:

struct dtv_property {
*u32 cmd;
__u32 reserved[3];
union {
__u32 data;
struct dtv_fe_stats st;
struct {
__u8 data[32];
__u32 len;
__u32 reserved1[3];
void reserved2;
} buffer;
} u;
int result;
} __attribute ((packed));

results in the D code:

struct dtv_property
{
__u32 cmd;
__u32[3] reserved;
union
{
__u32 data;
struct dtv_fe_stats
{
__u8 len;
dtv_stats[4] stat;
}
dtv_fe_stats st;
struct
{
__u8[32] data;
__u32 len;
__u32[3] reserved1;
void* reserved2;
}
}
int result;
}

which is compile error not only because of the __u32 and __u8 types, but because of the field data in the anonymous struct.

I do not have any answers, but I think this marker for the problems needs to be put on record.

Unable to compile with llvm 3.7

I've tried to compile the master with 3.7 but it gives me the following compile error.

dmd -O -m64 -L-ldl -m64 -c tango/math/Probability.d -ofobjs/tango/math/Probability.o tango/math/IEEE.d(858): Deprecation: use std.math.isNaN to deal with NaN operands rather than floating point operator '!<=' dmd -O -m64 -L-ldl -m64 -c tango/math/random/ExpSource.d -ofobjs/tango/math/random/ExpSource.o dmd -O -m64 -L-ldl -m64 -c tango/math/random/Kiss.d -ofobjs/tango/math/random/Kiss.o dmd -O -m64 -L-ldl -m64 -c tango/math/random/NormalSource.d -ofobjs/tango/math/random/NormalSource.o tango.math.BigInt is deprecated. Please use std.bigint instead. tango/io/vfs/ZipFolder.d(302): Error: function tango.io.vfs.ZipFolder.ZipSubFolder.toString cannot have an in contract when overriden function object.Object.toString does not have an in contract tango/io/vfs/ZipFolder.d(977): Error: function tango.io.vfs.ZipFolder.ZipFile.toString cannot have an in contract when overriden function object.Object.toString does not have an in contract dmd -O -m64 -L-ldl -m64 -c tango/math/random/Random.d -ofobjs/tango/math/random/Random.o dmd -O -m64 -L-ldl -m64 -c tango/math/random/Twister.d -ofobjs/tango/math/random/Twister.o dmd -O -m64 -L-ldl -m64 -c tango/math/random/Ziggurat.d -ofobjs/tango/math/random/Ziggurat.o build/Makefile:131: recipe for target 'objs/tango/io/vfs/ZipFolder.o' failed make: *** [objs/tango/io/vfs/ZipFolder.o] Error 1 make: *** Waiting for unfinished jobs.... ==> ERROR: A failure occurred in build().

Don't name anonymous enums

D allows them, so I had to search-and-replace to get the Anonymous_x names out.

Debian 2.1 release on Fedora Rawhide

I picked up the Debian binary, even though I am currently using Fedora Rawhide. I'll reboot to Debian Sid in a minute.

Rawhide has:

/usr/lib64/libbz2.so.1
/usr/lib64/libbz2.so.1.0.6
/usr/lib/libbz2.so.1
/usr/lib/libbz2.so.1.0.6

but not the libbz2.so.1.0 that dstep requires. I hacked a link and used LD_LIBRARY_PATH and I got the help message with an incorrect command line. I then tried a correct command line.

|> LD_LIBRARY_PATH=$HOME/lib.Linux.x86_64 dstep --version /usr/include/libdvbv5/dvb-fe.h
clang version 3.7.0 (tags/RELEASE_370/final)
Target: x86_64-redhat-linux-gnu
Thread model: posix

An unknown error occurred: dstep.core.Exceptions.DStepException@dstep/core/Exceptions.d(13): An unknown error occurred

??:? void dstep.driver.Application.Application.startConversion(immutable(char)[]) [0x589994]
??:? void dstep.driver.Application.Application.run() [0x5892a4]
??:? int dstack.application.Application.Application.releaseStart() [0x59fcf6]
??:? _Dmain [0x58c0b7]
??:? D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv [0x5b9dee]
??:? void rt.dmain2.d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x5b9d44]
??:? void rt.dmain2.d_run_main(int, char**, extern (C) int function(char[][])).runAll() [0x5b9daa]
??:? void rt.dmain2.d_run_main(int, char, extern (C) int function(char[][])).tryExec(scope void delegate()) [0x5b9d44]
??:? _d_run_main [0x5b9ca1]
??:? main [0x58c151]
??:? __libc_start_main [0x5f0847ff]

dstep.core.Exceptions.DStepException@dstep/core/Exceptions.d(13): An unknown error occurred

??:? void dstep.driver.Application.Application.startConversion(immutable(char)[]) [0x589994]
??:? void dstep.driver.Application.Application.run() [0x5892a4]
??:? int dstack.application.Application.Application.releaseStart() [0x59fcf6]
??:? _Dmain [0x58c0b7]
??:? D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv [0x5b9dee]
??:? void rt.dmain2.d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x5b9d44]
??:? void rt.dmain2.d_run_main(int, char**, extern (C) int function(char[][])).runAll() [0x5b9daa]
??:? void rt.dmain2.d_run_main(int, char, extern (C) int function(char[][])).tryExec(scope void delegate()) [0x5b9d44]
??:? _d_run_main [0x5b9ca1]
??:? main [0x58c151]
??:? __libc_start_main [0x5f0847ff]

Don't bother naming anonymous structs from a typedef

For example, this C typedef

typedef struct {
const char* name;
const char* signature;
void*       fnPtr;
} JNINativeMethod;

currently gets translated to

alias _Anonymous_0 JNINativeMethod;
struct _Anonymous_0
{
    const(char)* name;
    const(char)* signature;
    void* fnPtr;
}

It seems pointless to name the struct then alias it, as the different struct namespaces- which I was unaware of till I read this, always wondered why struct typedefs would repeat the name and were overly verbose- don't exist in D anyway.

I understand that it may be two steps in the translation process now, but it would be nice if you could combine the two in this case.

having problems installing dstep on ubuntu 14.04

I am attempting to build dub on Ubuntu 14.04, I have dmd v2.071.0, DUB version 0.9.25 and libclang 3.4.

When I enter $ dub build I get the output:

Linking...
/usr/bin/ld: cannot find -lclang
collect2: error: ld returned 1 exit status
--- errorlevel 1
dmd failed with exit code 1.

typedef and anonymous structs

Something like this:

typedef struct { ... } struct_name;

is a common pattern in c. It's just a way to rename an anonymous struct.

dstep split this in two different declaration

struct anonymous
{
}

and

alias anonymous struct_name;

Original anonymous struct will be never used and can't be called from C code, so I think this two declaration can be merged in a single named struct.

Handling advanced pre-processor macros.

I'm wondering how to handle 'advanced' pre-processor stuff. I have two problems. The first problem is libclang only parses an active branch of pre-processor condition. E.g.

#define FOO

#ifdef FOO
// This will appear in AST.
#else
// This won't appear in AST.
#endif

I suppose it isn't possible to implement translation of pre-processor that handles all possible cases, but most common cases should be doable. The only idea I have is to load the content of file we are translating and parse macros manually (possibly using clang to parse the code inside ifdefs) to some AST form, suitable for further processing. This should be perfectly doable. The next problem is a design/implementation one.

Currently there is a thin wrapper around c libclang API. I think it would be nice if pre-processor information (that we extract from file) are available through the same API. What I propose is to extend the classes (structs) like Cursor, SourceLocation, Token etc. in a way that they can contain data created by the user of the thin wrapper (currently an instances of that structs can only be created by libclang). Then, one could iterate through AST containing all the information in a transparent way, without worrying if the data comes from libclang or from the pre-processor parser. For example I would like Cursor to have a constructor that allows to create an instance of it from user-supplied data. E.g.

struct Cursor
{
    this(string spelling, CXCursorKind kind, SourceRange, ...);

    /+ ... +/
}

An alternative would be to add separate API and then interleave them in actual translator code, however it seems to be messy.

What do you think about it?