gtfierro/pundat

pundat

A data archiver and metadata query processor for BOSSWAVE

Documentation can be found in the wiki

To install:

From Source

Have Go installed

Run:

$ go get -u github.com/gtfierro/pundat
$ go install github.com/gtfierro/pundat

Polishing for Release

Check the issues, but also:

• fold in savepoint into pundat
• fold in management of archive requests into pundat
• pundat -> pd on command line?

pundat's Issues

Fold in savepoint

https://github.com/gtfierro/savepoint

• Fix the rmc command to only delete related archive requests: not everything
• update docs on archive requests
• simplify the API so make it easier to code these

Archive Request Browser

Administrative interface to look at:

• what archive requests is the archiver subscribed to?
  • probably dice this up a few ways: search by namespace
• is there an archive request for URI ?

Add command-line query

Need the scriptable kind where we can add a query on the command line and redirect the output to a file

Command Line Tool: Range Checker

Given a UUID or a URI, return the ranges of time you are permitted to access

Usage:

$ pundat range scratch.ns/abc/def/+
< range 1 >
< range 2 >

Time unit guessing in queries needs work?

Timestamp like 1450655999999 was not identified as nanoseconds.

Pundat does not respect the use of the "as" keyword in queries for timeunit conversion

This was an oversight; should be simple to implement

Query Documentation

Need more query examples in the wiki

Command-line Queries

Provide flag to interpret timestamps in returned values

Fix cache invalidation

10 minutes is a bit long. Can we invalidate the cache on a much shorter time interval so it more closely lines up with each new blockchain block? Around 15sec or so

Make names consistent

This project has been called a few things; make this less obvious

Command Line Tool: Permission checker/granter

Easy way to

• verify access to the archiver
• give access to the archiver

Proposed interface:

$ pundat check -k <key> -a <archiver>
Key <key> has access to archiver at <abc> until <time>

$ pundat check -k <key1> -a <archiver>
Key <key> does not have access to archiver at <abc>

$ pundat grant -k <key1> -a <archiver> -e <expiry>
Granting access on archiver <abc> to key <key1> for <expiry>

Progress bar logic is incorrect

https://github.com/gtfierro/pundat/blob/master/actions.go#L487

Need to wait until ALL DOTs are published. Not just the first one

Enable querying data + metadata simultaneously

select uuid, name, data in (now, now -5min) where ....

Seeing this bug in production

panic: reflect: reflect.Value.Set using unaddressable value

goroutine 2019 [running]:
reflect.flag.mustBeAssignable(0x16)
        /usr/local/go/src/reflect/value.go:228 +0x17d
reflect.Value.Set(0xca04e0, 0xc43163bf90, 0x16, 0xca04e0, 0x0, 0x16)
        /usr/local/go/src/reflect/value.go:1345 +0x2f
gopkg.in/vmihailenco/msgpack%2ev2.ptrDecoderFunc.func1(0xc431e09450, 0xca04e0, 0xc43163bf90, 0x16, 0xca04e0, 0xc43163bf90)
        /home/gabe/go/src/gopkg.in/vmihailenco/msgpack.v2/decode_value.go:104 +0x2a4
gopkg.in/vmihailenco/msgpack%2ev2.(*Decoder).DecodeValue(0xc431e09450, 0xca04e0, 0xc43163bf90, 0x16, 0xc43163bf90, 0x16)
        /home/gabe/go/src/gopkg.in/vmihailenco/msgpack.v2/decode.go:185 +0x8c
gopkg.in/vmihailenco/msgpack%2ev2.decodeInterfaceValue(0xc431e09450, 0xcff3a0, 0xc43163bfa0, 0x194, 0xcff3a0, 0xc43163bfa0)
        /home/gabe/go/src/gopkg.in/vmihailenco/msgpack.v2/decode_value.go:243 +0xe4
gopkg.in/vmihailenco/msgpack%2ev2.(*Decoder).DecodeValue(0xc431e09450, 0xcff3a0, 0xc43163bfa0, 0x194, 0xc43163bfa0, 0x194)
        /home/gabe/go/src/gopkg.in/vmihailenco/msgpack.v2/decode.go:185 +0x8c
gopkg.in/vmihailenco/msgpack%2ev2.(*Decoder).decode(0xc431e09450, 0xca04e0, 0xc43163bfa0, 0x50, 0xdb45c0)
        /home/gabe/go/src/gopkg.in/vmihailenco/msgpack.v2/decode.go:180 +0x11c
gopkg.in/vmihailenco/msgpack%2ev2.(*Decoder).Decode(0xc431e09450, 0xc428e25e50, 0x1, 0x1, 0x0, 0x40)
        /home/gabe/go/src/gopkg.in/vmihailenco/msgpack.v2/decode.go:66 +0x6f
gopkg.in/vmihailenco/msgpack%2ev2.Unmarshal(0xc42c0dd4e0, 0x1, 0x1, 0xc428e25e50, 0x1, 0x1, 0x413700, 0x7fd59c70f358)
        /home/gabe/go/src/gopkg.in/vmihailenco/msgpack.v2/decode.go:37 +0x1bb
github.com/immesys/bw2bind.(*MsgPackPayloadObjectImpl).ValueInto(0xc42990e0c0, 0xca04e0, 0xc43163bf90, 0x7fd59c70f358, 0xc42990e0c0)
        /home/gabe/go/src/github.com/immesys/bw2bind/pobasic.go:183 +0xbc
github.com/gtfierro/pundat/archiver.(*Stream).start.func2(0xc427b22dd0, 0x150d740, 0xc4200f8600, 0x150bb60, 0xc4200f8240)
        /home/gabe/go/src/github.com/gtfierro/pundat/archiver/stream.go:156 +0x266
created by github.com/gtfierro/pundat/archiver.(*Stream).start
        /home/gabe/go/src/github.com/gtfierro/pundat/archiver/stream.go:129 +0xb5

`_name` incorrectly cached

Finish implementing "distinct" queries

Add local subscription queues

We do a fair amount of processing on each received message, so its worth creating an internal queue per stream to provide some slack so that we don't get auto-unsubscribed by the designated router.

Next version of BOSSWAVE will contain some mitigation for this, but we should have some queue-awareness on the archiver side as well.

Add timeouts to client bindings

Really should be using golang's context to help manage timeouts, severed connections, etc

Removed nested "Metadata" tag

We get responses formatted like this:

{
  "UUID": "102a21a8-c74b-3684-98c9-3ec107d9a8d5",
  "Path": "r0uT8QOzVuRiWIQTHWnEKUAl-JOgPXmy0gGfWMwddGQ=/s.hamilton/00126d0700000016/i.temperature/signal/temperature",
  "Metadata": {
    "Building": "Gabe's Apartment",
    "Room": "Kitchen",
    "_name": "temperature"
  }
}

but the keys really aren't under a "Metadata" parent key. Need to get rid of that