ioquatix / ffi-clang Goto Github PK

View Code? Open in Web Editor NEW

43.0 8.0 36.0 451 KB

Ruby FFI bindings for libclang 3.4+.

License: MIT License

Ruby 98.28% C 0.44% C++ 1.27%

libclang ffi ruby cpp

ffi-clang's Introduction

FFI::Clang

A light-weight wrapper for Ruby exposing libclang. Works for libclang v3.4+.

Installation

Add this line to your application's Gemfile:

gem 'ffi-clang'

And then execute:

$ bundle

Or install it yourself as:

$ gem install ffi-clang

Usage

Traverse the AST in the given file:

index = Index.new
translation_unit = index.parse_translation_unit("list.c")
cursor = translation_unit.cursor
cursor.visit_children do |cursor, parent|
	puts "#{cursor.kind} #{cursor.spelling.inspect}"
	
	next :recurse 
end

Library Version

Due to issues figuring out which library to use, we require you to manually specify it. For example, to run the tests, with MacPorts llvm/clang 3.4, use the following:

LLVM_CONFIG=llvm-config-mp-3.4 rake

Contributing

We welcome contributions to this project.

Fork it.
Create your feature branch (git checkout -b my-new-feature).
Commit your changes (git commit -am 'Add some feature').
Push to the branch (git push origin my-new-feature).
Create new Pull Request.

Developer Certificate of Origin

This project uses the Developer Certificate of Origin. All contributors to this project must agree to this document to have their contributions accepted.

Contributor Covenant

This project is governed by the Contributor Covenant. All contributors and participants agree to abide by its terms.

ffi-clang's People

Contributors

Stargazers

Watchers

ffi-clang's Issues

How to build gem?

Sorry for the dumb question, but how does one build the ffi-clang gem? There isn't a rakefile that I see. I do see references to a gem called teapot, but its rubygem page doesn't link to any helpful documentation.

I would like to be able to build the gem locally with changes I have made, and then deploy it locally.

move repository to some organization

Sending pull request to forked repo is troublesome(by default it tries to send it to @jarib's repo) so I would like to send it to some organization like ffi-clang.

sometimes exceptions don't raise from visit_children

Sometimes exceptions don't raise:

require 'ffi/clang'

File.write('foo.c', "int main(){}")

index = FFI::Clang::Index.new
translation_unit = index.parse_translation_unit('foo.c', nil, [], opts={:detailed_preprocessing_record => true})
p "start"
translation_unit.cursor.visit_children do |cursor, parent|
  p [cursor.kind, cursor.spelling]
  if cursor.kind == :cursor_macro_definition
    p "raising"
    raise NameError
    p "what"
  end
  next :recurse
end
p "done"

output:

Clang version detected: [3, 5]
"start"
[:cursor_macro_definition, "__llvm__"]
"raising"
[:cursor_function, "main"]
[:cursor_compound_stmt, ""]
"done"

[BUG] Using angle brackets in comments adds extra newlines `\n` characters

Hi @ioquatix,

Thank you for writing these bindings! 👍

I'm trying to extract function comments.
However, whenever there are angle brackets in a comment, extra newline \n characters are being added.
The comments are distorted with those extra newline characters.

Here's an example:

test-header.h

#pragma once

/**
 * This is the description of the `test_function`.
 * Following option strings are supported:
 * -a,--arg1 <VALUE>
 * -b,--arg2 <VALUE>
 */
void test_function(const char* option);

ffi-clang-test.rb

require 'ffi/clang'

idx = FFI::Clang::Index.new
tu = idx.parse_translation_unit("test-header.h")
cr = tu.cursor
cr.visit_children do |cursor, parent|
    next :continue if cursor.comment.kind == :comment_null

    puts
    puts ">> cursor.comment.child.text:"
    puts "#{cursor.comment.child.text}"

    puts
    puts ">> cursor.comment.child.text.dump:"
    puts "#{cursor.comment.child.text.dump}"

    puts
    puts ">> cursor.raw_comment_text:"
    puts "#{cursor.raw_comment_text}"

    next :recurse
end

After running the script, here's the output:

$ ruby ffi-clang-test.rb
>> cursor.comment.child.text:
 This is the description of the `test_function`.
 Following option strings are supported:
 -a,--arg1 
<VALUE
>
 -b,--arg2 
<VALUE
>

>> cursor.comment.child.text.dump:
" This is the description of the `test_function`.\n Following option strings are supported:\n -a,--arg1 \n<VALUE\n>\n -b,--arg2 \n<VALUE\n>"

>> cursor.raw_comment_text:
/**
 * This is the description of the `test_function`.
 * Following option strings are supported:
 * -a,--arg1 <VALUE>
 * -b,--arg2 <VALUE>
 */

You can observe that there are extra newline characters \n added in the output of cursor.comment.child.text causing the broken output.

Could you please look into it?
Thank you!

PFA the sample header and Ruby script in compressed format:

ffi-clang-test-files.zip

New Apple LLVM Clang not supported

The new clang shipped with XCode 7 doesn't provide the original clang version info so the parsein utils.rb fails to get the version and stops.
Parsing the Apple version is possible but results in a different version of course - however the code seems to work fine.

if parts = clang_version_string.match(/(?:clang version|based on LLVM|Apple LLVM version) (\d+).(\d+)(svn)?/) <<
Do you plan to support the new Apple LLVM version?

I was having problems with llvm 10.0 on my linux desktop, I don't know if this addresses those issues.

Originally posted by @ioquatix in #49 (comment)

Better examples

The current solitary example shows well how to extract data/tokens, but it's very spartan, too spartan, to be honest.

Is there a chance to add some more extensive examples? Perhaps one showing AST rewriting (assuming that's possible via the C-api)?

Bindings Broken With Clang 16

The bindings don't work on clang 16 as can be seen here:

https://github.com/ioquatix/ffi-clang/actions/runs/8517036714/job/23326973558

The problem is:

Failure/Error: attach_function :is_explicit, :clang_CXXMethod_isExplicit, [CXCursor.by_value], :uint

I introduced this bug last year - although it looks like the Clang docs said this was available in 16 and were later changed (llvm/llvm-project@de4321c). Since I was testing with Clang 17, I didn't realize it broke clang 16.

C++ AST Introspection

I would like to add support for C++ AST introspection. See https://clang.llvm.org/doxygen/group__CINDEX__CPP.html#ga0c6580828662fd27cb7152e47c1b3e4e.

Would you prefer these to be added to clang/lib/cursor.rb or put in a separate file such as clang/lib/cxx_introspection.rb? Then I assume add methods to clang/cursor.rb?

visit_children SIGABRT

Reproducible with code which is kind of long, but nests visit_children calls.

(lldb) bt
* thread #1: tid = 0x2c0a0f, 0x00007fff90e1437a libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff90e1437a libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff8d9be8f7 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff9653762b libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000300f9 libruby.2.0.0.dylib`rb_bug + 185
    frame #4: 0x000000010013bb18 libruby.2.0.0.dylib`rb_thread_call_with_gvl + 99
    frame #5: 0x0000000102b559ca ffi_c.bundle`callback_invoke(cif=<unavailable>, retval=<unavailable>, parameters=<unavailable>, user_data=<unavailable>) + 372 at Function.c:479
    frame #6: 0x0000000102b5e517 ffi_c.bundle`ffi_closure_unix64_inner(closure=0x00000001002fc000, rvalue=0x00007fff5fbfe440, reg_args=0x00007fff5fbfe390, argp=0x00007fff5fbfe4a0) + 1223 at ffi64.c:629
    frame #7: 0x0000000102b5ed1e ffi_c.bundle`ffi_closure_unix64 + 70
    frame #8: 0x0000000102e0652a libclang.dylib`clang::cxcursor::CursorVisitor::Visit(CXCursor, bool) + 666
    frame #9: 0x0000000102e36066 libclang.dylib`bool clang::cxcursor::CursorVisitor::visitPreprocessedEntities<clang::PreprocessingRecord::iterator>(clang::PreprocessingRecord::iterator, clang::PreprocessingRecord::iterator, clang::PreprocessingRecord&, clang::FileID) + 438
    frame #10: 0x0000000102e0879b libclang.dylib`clang::cxcursor::CursorVisitor::visitPreprocessedEntitiesInRegion() + 923
    frame #11: 0x0000000102e07c8c libclang.dylib`clang::cxcursor::CursorVisitor::VisitChildren(CXCursor) + 1724
    frame #12: 0x0000000102e178e4 libclang.dylib`clang_visitChildren + 404
    frame #13: 0x0000000102b5eb9c ffi_c.bundle`ffi_call_unix64 + 76
    frame #14: 0x0000000102b5df19 ffi_c.bundle`ffi_call(cif=0x0000000102d49068, fn=0x0000000102e17750, rvalue=0x00007fff5fbfeed0, avalue=0x00007fff5fbfeeb0) + 1033 at ffi64.c:486
    frame #15: 0x0000000102b53650 ffi_c.bundle`rbffi_CallFunction(argc=<unavailable>, argv=<unavailable>, function=<unavailable>, fnInfo=<unavailable>) + 238 at Call.c:378
    frame #16: 0x0000000102b56f35 ffi_c.bundle`custom_trampoline(argc=<unavailable>, argv=<unavailable>, handle=<unavailable>) + 25 at MethodHandle.c:232
    frame #17: 0x0000000100135386 libruby.2.0.0.dylib`___lldb_unnamed_function2663$$libruby.2.0.0.dylib + 814
    frame #18: 0x0000000100134d67 libruby.2.0.0.dylib`___lldb_unnamed_function2662$$libruby.2.0.0.dylib + 614
    frame #19: 0x0000000100122553 libruby.2.0.0.dylib`___lldb_unnamed_function2526$$libruby.2.0.0.dylib + 11242
    frame #20: 0x000000010012d046 libruby.2.0.0.dylib`___lldb_unnamed_function2578$$libruby.2.0.0.dylib + 139
    frame #21: 0x000000010012dbc6 libruby.2.0.0.dylib`rb_iseq_eval_main + 138
    frame #22: 0x00000001000354c4 libruby.2.0.0.dylib`___lldb_unnamed_function568$$libruby.2.0.0.dylib + 128
    frame #23: 0x0000000100035415 libruby.2.0.0.dylib`ruby_run_node + 78
    frame #24: 0x0000000100000f26 ruby`main + 79
    frame #25: 0x00007fff9bf265c9 libdyld.dylib`start + 1

FFI::Clang::CodeCompletion::Results Finalizer Exception

Although the code completion tests complete successfully when run, they in fact are throwing exceptions in their finalizers.

For example:

C:/msys64/usr/local/ruby-3.3.0/bin/rspec: warning: Exception in finalizer #<FFI::AutoPointer::Releaser:0x0000021143e94020 @ptr=#<FFI::Pointer address=0x0000021145bf78d0>, @proc=#<Method: FFI::Clang::CodeCompletion::Results.release(pointer) C:/Source/ffi-clang/lib/ffi/clang/code_completion.rb:29>, @autorelease=true>
C:/msys64/usr/local/ruby-3.3.0/lib/ruby/gems/3.3.0/gems/ffi-1.16.4/lib/ffi/struct_by_reference.rb:58:in `to_native': wrong argument type FFI::Pointer (expected FFI::Clang::Lib::CXCodeCompleteResults) (TypeError)

        raise TypeError, "wrong argument type #{value.class} (expected #{@struct_class})"
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        from C:/Source/ffi-clang/lib/ffi/clang/code_completion.rb:31:in `dispose_code_complete_results'
        from C:/Source/ffi-clang/lib/ffi/clang/code_completion.rb:31:in `release'
        from C:/msys64/usr/local/ruby-3.3.0/lib/ruby/gems/3.3.0/gems/ffi-1.16.4/lib/ffi/autopointer.rb:160:in `call'
        from C:/msys64/usr/local/ruby-3.3.0/lib/ruby/gems/3.3.0/gems/ffi-1.16.4/lib/ffi/autopointer.rb:160:in `release'
        from C:/msys64/usr/local/ruby-3.3.0/lib/ruby/gems/3.3.0/gems/ffi-1.16.4/lib/ffi/autopointer.rb:151:in `call'
1 example, 0 failures, 1 passed
Finished in 0.3389161 seconds

The problem is the the pointer to Lib.dispose_code_complete_results(pointer) is a raw pointer but it should be a pointer to a CXCodeCompleteResults struct.

Releases to rubygems.org?

I noticed that the last release was 3 years ago while there are some improvements in the master branch. Do you install ffi-clang from GitHub and not use rubygems.org?

Won't load as-is on cygwin; very hacky fix included

This gem won't be able to be used as-is, since cygwin1.dll apparently does not export the structure of time_t.

Here is a potential fix, assuming LLVM+Clang have been installed into c:/Program Files (x86)/ (default installation path as defined by CMake):

require "ffi"
FFI.add_typedef :pointer, :time_t
FFI.typedef :pointer, :time_t
ENV['LIBCLANG'] = "c:/progra~2/LLVM/bin/libclang.dll"
require "ffi/clang"

Obviously the type definition of time_t here would unusable, but I also have not encountered an issue so far.

libclang version

Currently my branch is tested on MacPorts's clang 3.4(trunk) since I'm using http://clang.llvm.org/doxygen/group__CINDEX.html to add features.
Though most env uses 3.3 because it's the latest release.
So we need to decide which libclang to use.

But I want to note there is other solution: forking libclang
libclang's feature is way behind clang's C++ API and I don't want to wait the llvm's release cycle.
There is some patch that's useful but not merged(e.g http://www.mail-archive.com/[email protected]/msg43766.html ) too.
And we don't need to find libclang if we build libclang fork and install it with gem.

Strange Error

require 'ffi'
require 'ffi/clang'
class TestStruct < FFI::Struct
  layout :a,:int,
         :b,:int,
         :c,:pointer
end

ERROR Message:

Traceback (most recent call last):
	6: from t-ffistruct.rb:3:in `<main>'
	5: from t-ffistruct.rb:4:in `<class:TestStruct>'
	4: from /usr/local/lib/ruby/gems/2.6.0/gems/ffi-1.11.1/lib/ffi/struct.rb:218:in `layout'
	3: from /usr/local/lib/ruby/gems/2.6.0/gems/ffi-1.11.1/lib/ffi/struct.rb:306:in `array_layout'
	2: from /usr/local/lib/ruby/gems/2.6.0/gems/ffi-1.11.1/lib/ffi/struct.rb:266:in `find_field_type'
	1: from /usr/local/lib/ruby/gems/2.6.0/gems/ffi-1.11.1/lib/ffi/struct.rb:272:in `find_type'
/usr/local/lib/ruby/2.6.0/mkmf.rb:1255:in `find_type': wrong number of arguments (given 1, expected 2+) (ArgumentError)

Remove: require 'ffi/clang', no error anymore.

AST failure because wrong clang compiler is used in test

In spec/clang/index_spec.rb there is a problem with the #create_translation_unit test. Because different versions of llvm and clang may be installed on the host, the AST generated by clang may not be suitable for the version of libclang being used.

The error output is as follows:

> rake --trace       
** Invoke default (first_time)
** Invoke spec (first_time)
** Execute spec
/Users/samuel/.rvm/rubies/ruby-2.0.0-p247/bin/ruby -S rspec ./spec/clang/comment_spec.rb ./spec/clang/cursor_spec.rb ./spec/clang/diagnostic_spec.rb ./spec/clang/file_spec.rb ./spec/clang/index_spec.rb ./spec/clang/source_location_spec.rb ./spec/clang/source_range_spec.rb ./spec/clang/token_spec.rb ./spec/clang/translation_unit_spec.rb ./spec/clang/type_spec.rb ./spec/clang/utils_spec.rb
Clang version detected: [3, 4]
Run options: exclude {:upto_3_4=>true, :upto_3_3=>true, :upto_3_2=>true}
..............................................................................................................................................................F......................................................................................

Failures:

  1) FFI::Clang::Index#create_translation_unit can create translation unit from a ast file
     Failure/Error: tu = index.create_translation_unit "#{TMP_DIR}/simple.ast"
     FFI::Clang::Error:
       error parsing "/Users/samuel/Documents/Programming/Scripting/ffi-clang/spec/tmp/simple.ast"
     # ./lib/ffi/clang/index.rb:49:in `create_translation_unit'
     # ./spec/clang/index_spec.rb:61:in `block (3 levels) in <top (required)>'

Finished in 0.51196 seconds
245 examples, 1 failure

Failed examples:

rspec ./spec/clang/index_spec.rb:59 # FFI::Clang::Index#create_translation_unit can create translation unit from a ast file
/Users/samuel/.rvm/rubies/ruby-2.0.0-p247/bin/ruby -S rspec ./spec/clang/comment_spec.rb ./spec/clang/cursor_spec.rb ./spec/clang/diagnostic_spec.rb ./spec/clang/file_spec.rb ./spec/clang/index_spec.rb ./spec/clang/source_location_spec.rb ./spec/clang/source_range_spec.rb ./spec/clang/token_spec.rb ./spec/clang/translation_unit_spec.rb ./spec/clang/type_spec.rb ./spec/clang/utils_spec.rb failed

Solution: If LLVM_CONFIG was specified, should somehow be used to find the compiler?

visit_children with :recurse doesn't visit every child

I must be missing something. Using the code example in the readme on a small snippet of C++ code doesn't print anything, and trying on a larger code sample only prints a very small subset of the nodes in the AST. Here's the small chunk of C++ code I'm trying to parse:

ofPtr< ABTest > ABTest::create()
{
  ofPtr< ABTest > create( new ABTest() );
  create->init( create, ABTest::getValueStoreType() );
  if (create->_this)
  {
    return create;
  }
  return ofPtr< ABTest >();
}

Here's the ruby code that's trying to parse it:

require 'ffi/clang'

include FFI::Clang

index = Index.new(false)
translation_unit = index.parse_translation_unit("/Users/cameron/Desktop/test.cpp")
cursor = translation_unit.cursor

cursor.visit_children do |cursor, parent|
  puts "#{cursor.kind} #{cursor.spelling.inspect}"
  next :recurse
end

This code prints nothing. Is it possible next :recurse isn't working for some reason? Thanks for your help :)

Why does Lib.bitmask_from take a Hash?

Lib.bimask_from takes a hash (the opts parameter) but does not use the hash value:

https://github.com/ioquatix/ffi-clang/blob/main/lib/ffi/clang/lib.rb#L69

The test show the following cases work:

index.parse_translation_unit(fixture_path("a.c"), [], [], [:incomplete, :single_file_parse, :cache_completion_results])
index.parse_translation_unit(fixture_path("a.c"), [], [], {:incomplete => 654, :single_file_parse => 8, :cache_completion_results => 93})

However, Lib.bimask_from only accepts values that are defined in the enum passed to it (in this case Lib::TranslationUnitFlags). Thus the hash seems pointless - and at least to me very non-intuitive.

But maybe this can't change for backwards compatibility reasons?

What do I have to require?

Sorry for stupid question, I am not really a Ruby programmer:

I am doing this:

require "rubygems"
require "ffi/clang/lib"

And I am getting the following error:

/var/lib/gems/1.8/gems/ffi-clang-0.2.0/lib/ffi/clang/lib.rb:25: uninitialized constant FFI::Library (NameError)
from /usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:59:in `gem_original_require'
from /usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:59:in `require'
from ./test.rb:3

Should I require some other file?

uninitialized constant FFI::Clang::Lib::TranslationUnitFlags