redsift/rkdb

kdb+ Rust embedding and IPC

rkdb

kdb+ bindings for Rust. Originally forked from and substantially based on the Krust cargo package by adwhit via robsmith11.

These bindings enable Rust to be used as inside Q to add additional functionality. More generally they also enable Rust to communicate with kdb+ in a memory-safe way.

In addition to Krust, this variant features:

• Support for macOS
• Support for the GUID type and sd0x function call
• Fix for kerror strings when exported from Rust to Q
• Fixes for embedding
• Performance features
• Dynamic export for symbols to be weak linked
• Helpers for querying attributes on types

This variant incorporates robsmith11 updates for the current rust nightly.

For an example of how to embed Rust code within Q, see demos/embed. For an example of how to perform IPC between Rust and Q, see demos/ipc.

Building for Embedding

When built for embedding, the symbols are provided by the hosting Q process and the library must not include the duplicated functions. This requires weak linking which currently needs to be manually archived on macOS by specifying all the missing symbols that should be ignored at link time. This symbols are generated at build time as the constant rkdb::SYMBOLS and is currently used by setting .cargo/config in the library target to:

[target.'cfg(target_os="macos")']
rustflags = ["-Clink-args=-Wl,-U,_ktn -Wl,-U,_knk -Wl,-U,_ku -Wl,-U,_ka -Wl,-U,_kb -Wl,-U,_kg -Wl,-U,_kh -Wl,-U,_ki -Wl,-U,_kj -Wl,-U,_ke -Wl,-U,_kf -Wl,-U,_kc -Wl,-U,_ks -Wl,-U,_kd -Wl,-U,_kz -Wl,-U,_kt -Wl,-U,_ktj -Wl,-U,_kp -Wl,-U,_kpn -Wl,-U,_xT -Wl,-U,_xD -Wl,-U,_ktd -Wl,-U,_ss -Wl,-U,_sn -Wl,-U,_ymd -Wl,-U,_dj -Wl,-U,_setm -Wl,-U,_r1 -Wl,-U,_r0 -Wl,-U,_m9 -Wl,-U,_sd1 -Wl,-U,_sd0 -Wl,-U,_sd0x -Wl,-U,_k -Wl,-U,_dl -Wl,-U,_ja -Wl,-U,_js -Wl,-U,_jk -Wl,-U,_jv -Wl,-U,_krr -Wl,-U,_orr -Wl,-U,_dot -Wl,-U,_okx -Wl,-U,_b9 -Wl,-U,_d9"]

Example

Reference kdb-rs-hash for an example of embedding.

Building for IPC

To use this library for IPC, you will first need to compile a static library from the kx-supplied object file c.o using ar. The static library should be placed in src/c/libkdb.a.

ar rcs libkdb.a c.o
mv libkdb.so src/c

c.o can be found here.

Compatibility

• 64 bit i.e. commercial version of Q >= 3.6.
• macOS (with XCode >= 9.4.1) and Linux.
• rustc >= 1.31.0

License

Based substantially on adwhit and licensed under the same MIT license.

rkdb's Issues

cannot pass symbols from kdb to rust function

Looks like cannot pass list of symbols from kdb to rust function

#[no_mangle]
pub extern "C" fn ksymbols(k: *const K) -> *const K {
    match KVal::new(k) {
        KVal::Symbol(KData::List(s)) => println!("Symbols received {:?}", s),
        _ => println!("no symbols"),
    }
    kvoid()
}

q)ksymbols[`a`b`c`d]
thread '<unnamed>' panicked at 'byte index 80 is not a char boundary; it is inside '\u{0}' (bytes 79..80) of `aC[��������lj��
��������ljfN�Z�5`����������aj>gGf��p����������.Q.fl�eGf��������.Q.ff�^���������.Q.ajf0����������g����������`[...]', src/libcore/str/mod.rs:2154:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
q: warning: q crashed, killed by SIGABRT.
q itself has not crashed, but for transparency,
it will now kill itself (without dumping core) with the same signal

Unit tests and CI integration required

• CI integration
• Embed unit tests
• IPC unit tests

Failure parsing table

Hi Guys,

Apologies if the issue I am reporting turns out not to be an issue!

I have been trying to leverage your project as a layer on top of rust avro lib "avro_rs". End goal is to be able to parse avro messages coming over kafka from within kdb. I am able to encode/decode mixed lists successfully but encountered following issue while trying to parse a table.

Here is my simple rust function
pub extern "C" fn encode_avro_tbl(k: *const K) -> () { match KVal::new(k) { KVal::Table(b) => println!("Found table {:?}", b), _ => println!("No match") }; }

After I build this and try to use in q, I get following seg fault

q)encode: librkdbavro 2:(encode_avro_tbl;1) q)t:([]sid:5?100i; sym:5?2;price:5?10f;size:5?100)
q)t
sid sym price size

88 pl 7.85033 12
58 nn 5.347096 10
68 gl 7.111716 1
45 gk 4.11597 90
2 bg 4.931835 73
q)
q)meta t

c	t f a
sid	i
sym	s
price	f
size	j
q)
q)encode[t]
thread '' panicked at 'byte index 88 is not a char boundary; it is inside '\u{0}' (bytes 87..88) of `sid

                                                                                                             P)�9
                                                                                                                 ��**�E`���`E`���F`������`[...]', src/libcore/str/mod.rs:2133:5

stack backtrace:
0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
1: core::fmt::write
2: std::io::Write::write_fmt
3: std::panicking::default_hook::{{closure}}
4: std::panicking::default_hook
5: std::panicking::rust_panic_with_hook
6: rust_begin_unwind
7: core::panicking::panic_fmt
8: core::str::slice_error_fail
9: core::str::traits::<impl core::slice::SliceIndex for core::ops::range::Range>::index::{{closure}}
10: ::fmt
11: core::fmt::builders::DebugInner::entry
12: core::fmt::builders::DebugList::entry
13: <&T as core::fmt::Debug>::fmt
14: core::fmt::builders::DebugTuple::field
15: <&T as core::fmt::Debug>::fmt
16: core::fmt::builders::DebugTuple::field
17: <rkdb::kbindings::KVal as core::fmt::Debug>::fmt
18: core::fmt::builders::DebugTuple::field
19: <rkdb::kbindings::KVal as core::fmt::Debug>::fmt
20: core::fmt::write
21: <std::io::stdio::Stdout as std::io::Write>::write_fmt
22: std::io::stdio::_print
23: encode_avro_tbl
24: dl
25: setSysNotify
26: setSysNotify
27: setSysNotify
28: setSysNotify
29: setSysNotify
30: m9
31: knt
32:
33: sd1
34: sd1
35: dl
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 5
q: warning: q crashed, killed by SIGABRT.
q itself has not crashed, but for transparency,
it will now kill itself (without dumping core) with the same signal
`

Is it an issue or am I using it incorrectly? I would really appreciate any help.

Thanks,
Ajay

Update code for doc.rs generation

Currently, very little documentation is produced.

Invalid deserialization?

kdb callback not working

Hi,
I am trying to call a q function from within a rust function as follows. I basically added this function to the embed crate in demo

Running on macos

#[no_mangle]
pub extern "C" fn test_callback(_: *const K) -> *const K {
    println!("test_callback");
    unsafe {
            for i in 1..5 {
                println!("looping");
                k(0, ffi::CString::new("callback".as_bytes().to_vec()).unwrap().as_ptr(), 0);
            }
    }
    kvoid()
}

When i call it from q, it does not seem to call "callback" function

q)test_callback: `libembed 2:(`test_callback;1)
q)callback:{[x] show "callback..."; show x}
q)test_callback[`]
test_callback
looping
looping
looping
looping
q)

Is it something not supported yet? This is a minitiarized version of what I am trying to do.

Verify module layout is sane

Was originally reworked to simplify dynamic symbol extraction but choice of names may be sub optimal. Review based on ongoing binding work.