sv / kdbgo Goto Github PK
View Code? Open in Web Editor NEWkdb+ client driver for Go
License: MIT License
kdb+ client driver for Go
License: MIT License
Some of the v0 api is not very familiar to regular users of net package and can be made more modular.
v1beta branch uses more straightforward Conn and Dial for managing connections
Im trying to use UnmarshalDict any i got it to work with all fields but a list of char
My table structure
c | t f a
-----------| -----
id | s
dispname | C
order | i
update_time| z
My Struct for this table
type Linkgroup struct {
Id string `json:"id"`
Dispname string `json:"dispname"`
Order int32 `json:"order"`
Update_time time.Time `json:"update_time"`
}
i do a db.Call("0!select from linkgroups")
Then i do the following
tbl := t.Data.(kdb.Table)
dict := tbl.Index(2)
var data2 models.Linkgroup
err := kdb.UnmarshalDict(dict, data2)
fmt.Println(data2)
All the fields work except Dispname.
Thank you for any help, hopefully you still look at this repo
Here is my code, please check it out.
package main
import (
"bytes"
"fmt"
"log"
"os"
"os/exec"
"strconv"
kdb "github.com/sv/kdbgo"
)
var testPort = 0
func StartQ(ready chan struct{}) {
fmt.Println("Starting q process on random port")
_, err := exec.LookPath("q")
if err != nil {
log.Fatal("installing q is in your future")
}
cmd := exec.Command("q", "-p", "0W", "-q")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal("Failed to connect stdin", err)
}
stdin.Write([]byte(".z.pi:.z.pg:.z.ps:{value 0N!x};system\"p\"\n"))
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Fatal("Failed to connect stdout", err)
}
stderr, err := cmd.StderrPipe()
if err != nil {
log.Fatal("Failed to connect stderr", err)
}
err = cmd.Start()
if err != nil {
log.Fatal("Failed to start q", err)
}
buf := make([]byte, 16)
n, err := stdout.Read(buf)
if err != nil {
log.Fatal("Failed to fill port number for child process: ", err)
}
testPort, err = strconv.Atoi(string(buf[:bytes.IndexByte(buf, 'i')]))
if err != nil {
fmt.Println("Failed to setup listening port", string(buf[:n]), err)
cmd.Process.Kill()
os.Exit(1)
}
fmt.Println("Listening port is ", testPort)
go func() {
for {
buf := make([]byte, 256)
n, err := stderr.Read(buf)
fmt.Println("Q stderr output:", string(buf[:n]))
if err != nil {
fmt.Println("Q stderr error:", err)
return
}
}
}()
go func() {
for {
buf := make([]byte, 256)
_, err := stdout.Read(buf)
//fmt.Println("Q stdout output:", string(buf[:n]))
if err != nil {
fmt.Println("Q stdout error:", err)
return
}
}
}()
ready <- struct{}{}
cmd.Wait()
//stdin.Write([]byte("show `exiting_process;\nexit 0\n"))
}
func main() {
ready := make(chan struct{})
go StartQ(ready)
<-ready
log.Println("Connect to kdb+ through unix socket.")
con, err := kdb.DialUnix("localhost", testPort, "")
if err != nil {
panic(err)
}
log.Println("Close the connection")
err = con.Close()
if err != nil {
panic(err)
}
}
Hi, we are trying to make multiple select queries in parallel to retrieve data from kdb.
query:- select open:first price, high:max price, low:min price, close:last price, volume:sum volume by sym from trade where time within time1 time 2
we are encountering an error : "Slice bounds out of range" from Decode function.
panic: slice bounds out of range
goroutine 106 [running]:
bufio.(*Reader).Read(0xc000049380, 0xc00112d328, 0x8, 0x8, 0xc00f669c08, 0x441cd3, 0x8)
C:/Go/src/bufio/bufio.go:234 +0x3ce
io.ReadAtLeast(0xe339e0, 0xc000049380, 0xc00112d328, 0x8, 0x8, 0x8, 0x5, 0xc000dc7560, 0x0)
C:/Go/src/io/io.go:310 +0x8f
io.ReadFull(...)
C:/Go/src/io/io.go:329
encoding/binary.Read(0xe339e0, 0xc000049380, 0xe4a140, 0x1487ec0, 0xbd22c0, 0xc00112d2f8, 0x0, 0xc00f669e28)
github.com/sv/kdbgo.Decode(0xc000049380, 0xc00011c270, 0xc00011c201, 0xc000621120, 0x0)
This error occurs only when there is huge data for processing.
We don't see this error when there is only one select query.
Could some one please look into this ? and recommend a fix.
Environment
kdb windows 32bit
go version go1.12.7 windows/amd64
OS: windows
With 64-bit personal edition installed, "go test -v" failed at line 47 in kdb_test.go.
Error message:
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
github.com/sv/kdbgo.TestMain(0xc0000bc200)
/home/even/Experiments/kdb/kdbgo/kdb_test.go:47 +0x813
main.main()
_testmain.go:80 +0x13e
exit status 2
Hey, I noticed that the function UnmarshalDict
have some behaviour around field names that may be undesirable (certainly is for my usecase).
The main issue is that when it looks up a fieldname it will make the first character uppercase if necessary, but no further changes. That means that if a dict field uses snake case, like this:
this_is_a_field
the corresponding field in a struct would have to be:
type MyStruct struct {
This_is_a_field int
}
Which both brake go conventions and is also confusing.
For my own purpose I rewrote the function the following way:
func unmarshalResponse(t kdb.Dict) (*OrderResponse, error) {
var (
keys = t.Key.Data.([]string)
vals = t.Value.Data.([]*kdb.K)
v = new(OrderResponse)
)
vv := reflect.Indirect(reflect.ValueOf(v))
for i := range keys {
val := vals[i].Data
fv := vv.FieldByName(strcase.ToCamel(keys[i]))
if !fv.IsValid() {
return nil, fmt.Errorf("field does not exist %q", keys[i])
}
if fv.CanSet() && reflect.TypeOf(val).AssignableTo(fv.Type()) {
fv.Set(reflect.ValueOf(val))
}
}
return v, nil
}
It uses a thirdparty library to convert the name to CamelCase, but that could be done locally, I was just being lazy.
(an added benefit with this version is that it will error out if a correct struct field can not be found).
In addition it might be beneficial to support tags as an alternative to strict field naming similar to how the json marshalling works, so it would be possible to do something along these lines.
type MyStruct struct {
Foo int `kdb:"this_is_a_field"`
}
I'm happy to write up a PR if you think this would be beneficial.
I'm also working on functionality to Marshal structs and maps into Dicts that might be useful as well.
I couldn't find a description how the project is licensed. It would be good to provide this information.
In https://github.com/sv/kdbgo/blob/master/kdb.go#L208 the port, which is an int value is casted to a string by string(port)
.
This is not correct.
Consider using strconv.Itoa(port)
or fmt.Sprintf.
panic: runtime error: slice bounds out of range
goroutine 162 [running]:
bufio.(*Reader).Read(0xc4202181e0, 0xc4216b5c88, 0x8, 0x8, 0x8, 0x16c96e0, 0x0)
/usr/local/go/src/bufio/bufio.go:227 +0x3bf
io.ReadAtLeast(0x1888be0, 0xc4202181e0, 0xc4216b5c88, 0x8, 0x8, 0x8, 0x16c9760, 0x1767701, 0xc4216b5c88)
/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x1888be0, 0xc4202181e0, 0xc4216b5c88, 0x8, 0x8, 0x8, 0xc420036f00, 0xc420070218)
/usr/local/go/src/io/io.go:327 +0x58
encoding/binary.Read(0x1888be0, 0xc4202181e0, 0x1890300, 0x1cb5b00, 0x1708000, 0xc4216b5348, 0xc420704380, 0xc4216efad0)
/usr/local/go/src/encoding/binary/binary.go:245 +0xf48
github.com/sv/kdbgo.Decode(0xc4202181e0, 0xc420222028, 0x1, 0xc421213f40, 0x0)
/github.com/sv/kdbgo/decode.go:141 +0x95
github.com/sv/kdbgo.(*KDBConn).Call(0xc4201e2230, 0xc424473dc0, 0x3d, 0x0, 0x0, 0x0, 0xc42000e018, 0xc420e1bed8, 0x1)
/github.com/sv/kdbgo/kdb.go:85 +0x179
I just started looking into this as I've been reaching a segmentation fault in kdb+ when passing strings over IPC (via AsyncCall, Call, and WriteMessage). The issue seems to lie in the Compress function which appears to truncate some input objects. When I remove compression from the Encode function, the object is sent as expected. I will upload more details later, but when I try passing several strings over IPC, each greater than 800 characters, I receive the seg fault in kdb+.
time:= &kdb.K{kdb.KN, kdb.NONE, []int64{t2.Nanoseconds()}}
when I use time.Duration to upsert KDB I got the
ERROR: logging before flag.Parse: E0223 11:52:50.294900 13024 decode.go:129] Failed to read message header: EOF
Alternatives: https://gof.rs/projects/uuid/ or https://github.com/google/uuid
reason: package is not maintained/versioned and has some issues nu7hatch/gouuid#26
Can Decode also be used to decode data from disk, or is this a different format?
I am thinking of addressing the installing q is in your future
to install Q on Travis CI since I believe some of the tests are failing when run with Q installed on my machine
I was thinking we could check in the 32 bit binary of Q into the repository and run that. Do you have any approach in mind?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.