distributedclocks / govector Goto Github PK
View Code? Open in Web Editor NEWVector clock logging library for Go
License: MIT License
Vector clock logging library for Go
License: MIT License
Formulate the subscriber API version .1 This should include authentication/nonce bits for setting up the connection. This API should reflect client concerns. Some things that it may contain are: GetOldEvents (before subscriber joiner), CreateFilter (only notify subscriber of events of particular nature), RemoveFilter (disable the filtering), PauseStream (pause the publishing of events to this subscriber), UnPauseStream (resume publishing of events to this subscriber -- should events that were on pause be sent?).
A complete API would need to be specified as a JSON RPC set of method calls, and an example client (in Go, or JS) that would interface with the server-side API.
Hello,
Say I am very pleased with the logging and believe everything works perfectly. I don't want to change my code but I now longer want to log large number of files for my system.
Is there a method/flag in the library to disable logging completely? I see there are some to print out the logging to the screen, but none to disable.
Both publishers and subscribers are being served by the same RPC server (the default), this may mean that a subscriber can make publisher RPC calls and vice versa.
Create separate servers so that only publisher RPC calls are available on the publisher port and only subscriber rpc calls are available on the subscriber port.
Regex filters on PID, VClock, message receipt time.
Local message compression filter to compress a sequence of local messages from a single host/PID
When a publisher disconnects it has maybe dropped out of the distributed system. It would be good to log that as an event with an appropriate vector clock entry.
The vector clock could either be reconstructed from the most recent message received or messages could be refactored to record which publisher they came from if we wanted to record the last seen vclock.
Replace custom parsing with the flag package.
Issue:
When I have several clients logging events to the same log file, only the first client of the library has an entry with a vector clock starting at 1. The rest of the clients do not have an entry where the vector clock starts at 1 for them. As a result, when I upload a log file to https://bestchai.bitbucket.io/shiviz/ I get an error saying
Logical clock values for each host must start at 1.
The clock value for the first event for host "34784" is 2.
Code used to obtain the problem:
This is the structure of my project
These extensions of these files should be changed to '.go'
client.txt
network.txt
server.txt
# How to run this project
$ server.go # in one terminal, leave it running
$ client.go <IP:port of the server> # run this several times, in different terminals
Proposed fix: If the log file exists, we still need to log some kind of event, which will result in having an entry with a vector clock starting at 1.
patchFix.txt
Update: Actually I don't know enough about the project to be sure that this will work. In particular, Im not sure why we have different execution numbers, and do entries for new processes must start with vector clock 1?
Code I am using:
package main
import (
"github.com/DistributedClocks/GoVector/govec"
)
func main() {
LoggerClient := govec.InitGoVectorMultipleExecutions("2", "test")
LoggerClient.PrepareSend("Sending", "blabla")
}
Problem:
I get the following error
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12dd1ad]
goroutine 1 [running]:
github.com/DistributedClocks/GoVector/govec.(*GoLog).PrepareSend(0xc420134100, 0x1393dde, 0x7, 0x1316880, 0x13d6600, 0x0, 0xc420125f70, 0x12de28e)
/Users/annygakh/go/src/github.com/DistributedClocks/GoVector/govec/govec.go:507 +0x25d
main.main()
/Users/annygakh/go/src/test-shiviz/test.go:12 +0x93
exit status 2
Possible fix:
It seems like there is missing initialization of private fields gv.encodingStrategy (and gv.decodingStrategy). The fields are initialized in InitGoVector, but not in InitGoVectorMultipleExecutions (which is what I am using). Attached is a proposed fix.
Refactor logging with a new method that checks the appropriate flags and then logs the requested level of information.
Must also update the Broker initialization in broker.go to accept debugging flags to turn on/off log messages/file logging.
Need to wrap the ServeConn into a go routine so that you can (1) service multiple subscribers, and (2) do this asynchronously.
Example of doing this with RPC over a TCP connection:
https://www.artima.com/weblogs/viewpost.jsp?thread=333589
Hi!
I've used this library in college for a little proyect and all the teammates I meet had the same problem I had.
In the govec program, inside the govec folder, line 208 has a bug where you passed 3 arguments to the function dec.Decode(). An error pop up saying how it's supposed to be and, actually, in the line 183 is correctly used. So i changed it to err = dec.Decode(&d.Payload) and it works.
I see is an easy fix, but using an IDE (for example Goland) it warns you and tries to prevent you from changing any file that is an external library. Maybe, for an even more amateur programmer than me, this could be the end of the work for a while.
Formulate the publisher API version .1 This should include authentication/nonce bits for setting up the connection. It should also include a set of event types that are relevant to logging in a distributed context: LogEvent, JoinSystem (implicit?), LeaveSystem, others?
A complete API would need to be specified as a JSON RPC set of method calls, and an example client that would interface with the server-side API.
Refactor logging with a new method that checks the appropriate flags and then logs the requested level of information.
There are a number of if statements checking flags that should be refactored in this way.
Refactor GoPublisher to its own .go file and update govec.go accordingly.
Index.html was modeled on a chat server. Should be simplified and updated to reflect current usage.
GoLog's initialize functions (Initialize and InitializeMutipleExecutions) do not accept parameters for GoVec settings.
Create a struct that holds the appropriate settings (gv.printonscreen (ShouldYouSeeLoggingOnScreen) gv.VConWire (ShouldISendVectorClockonWire),gv.debugmode (Debug), gv.realtime = false etc), modify the initialize functions to accept this struct and modify the GoLog struct to store and use the configuration struct.
Formulate the subscriber API version .1 This should include authentication/nonce bits for setting up the connection. It should also include a set of event types that are relevant to advertising set of system events to the client. Roughly this include the same set of events as in the publisher API: NewEvent, ProcessJoined, ProcessLeft,
A complete API would need to be specified as a JSON RPC set of method calls, and an example client (in Go, or JS) that would interface with the server-side API.
Write tests for an end to end application of the system. Currently, all we have is the code found in Client Server code found in /example.
Currently a single broker is used for the test suite, this may cause state based test failures. If a broker could be initialized to listen on different ports for each test then the tests will be better encapsulated.
Possible issue with knowing which port to connect to as data is stored in a suite variable. Tests do not appear to be run sequentially. This seems to be a function of the go testing package rather than go-check.
The broker will need to be parameterized in a number of ways:
Instead of architecting a command line madness, it would be great to push these into a config file and use an existing config file library for reading these options.
Sending an interface over vrpc causes a crash. For example the following types
type Operation interface {
OperationType() OperationType
GetMinerID() string
GetFileName() string
}
type AppendOperation struct {
Timestamp time.Time
Filename string
Record Record
RecordNum uint16
MinerId string
}
Where we are sending and expecting Operation. This causes deserialization to fail with the following message:
panic: reflect.Set: value of type map[string]interface {} is not assignable to type blockchain.Operation
goroutine 5 [running]:
reflect.Value.assignTo(0x74a1c0, 0xc00040c2d0, 0x15, 0x7b4559, 0xb, 0x75cf60, 0xc0003ee1d0, 0xc00040c2d0, 0x0, 0x0)
C:/Go/src/reflect/value.go:2239 +0x453
reflect.Value.Set(0x75cf60, 0xc0003ee1d0, 0x194, 0x74a1c0, 0xc00040c2d0, 0x15)
C:/Go/src/reflect/value.go:1373 +0xae
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).interfaceValue(0xc00029e700, 0x75cf60, 0xc0003ee1d0, 0x194, 0x535792, 0xc00029e700)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:284 +0x163
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.decodeInterfaceValue(0xc00029e700, 0x75cf60, 0xc0003ee1d0, 0x194, 0xc0003ee1d0, 0x194)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:256 +0x1ea
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.ptrDecoderFunc.func1(0xc00029e700, 0x71eb20, 0xc0003ee1d0, 0x16, 0x71eb20, 0x0)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:124 +0xca
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).DecodeValue(0xc00029e700, 0x71eb20, 0xc0003ee1d0, 0x16, 0xc0003ee1d0, 0x16)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:224 +0x93
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.decodeInterfaceValue(0xc00029e700, 0x748420, 0xc00040c288, 0x194, 0x748420, 0x749800)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:267 +0xaa
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).DecodeValue(0xc00029e700, 0x748420, 0xc00040c288, 0x194, 0xc00040c288, 0x194)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:224 +0x93
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).decode(0xc00029e700, 0x7216e0, 0xc00040c288, 0xc0003ea110, 0xe)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:212 +0x182
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).Decode(0xc00029e700, 0xc00006f9d0, 0x1, 0x1, 0x0, 0x763718)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:98 +0x73
github.com/DistributedClocks/GoVector/govec.(*VClockPayload).DecodeMsgpack(0xc00040c270, 0xc00029e700, 0xc00040c270, 0x2b70338)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/govec/govec.go:184 +0xcf
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.decodeCustomValue(0xc00029e700, 0x7636e0, 0xc00040c270, 0x16, 0xc00040c270, 0x16)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:174 +0x118
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.decodeCustomValueAddr(0xc00029e700, 0x772fe0, 0xc00040c270, 0x199, 0x772fe0, 0x452130)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:132 +0x7c
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).DecodeValue(0xc00029e700, 0x772fe0, 0xc00040c270, 0x199, 0xc00040c270, 0x199)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:224 +0x93
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).decode(0xc00029e700, 0x7636e0, 0xc00040c270, 0x0, 0x40)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:212 +0x182
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).Decode(0xc00029e700, 0xc00006fcc8, 0x1, 0x1, 0xc000036000, 0xc00006fcb8)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:98 +0x73
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.Unmarshal(0xc000286060, 0x52, 0x52, 0xc00006fcc8, 0x1, 0x1, 0x30, 0x772fe0)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:36 +0xb1
github.com/DistributedClocks/GoVector/govec.defaultDecoder(0xc000286060, 0x52, 0x52, 0x7636e0, 0xc00040c270, 0x71eaa0, 0xc0000fe3e0)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/govec/govec.go:371 +0x85
github.com/DistributedClocks/GoVector/govec.(*GoLog).UnpackReceive(0xc000108080, 0x7b886f, 0x14, 0xc000286060, 0x52, 0x52, 0x71eb20, 0xc0003ee1d0, 0x1)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/govec/govec.go:553 +0x102
github.com/DistributedClocks/GoVector/govec/vrpc.(*RPCServerCodec).ReadRequestBody(0xc000202080, 0x71eb20, 0xc0003ee1d0, 0x71eb20, 0xc0003ee1d0)
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/govec/vrpc/vrpc.go:138 +0xee
net/rpc.(*Server).readRequest(0xc0000943c0, 0x814ea0, 0xc000202080, 0xc00003c360, 0xc00003c3c0, 0xc00042df88, 0x6f897a, 0x5040408, 0xc0000524a0, 0x20, ...)
C:/Go/src/net/rpc/server.go:569 +0x299
net/rpc.(*Server).ServeCodec(0xc0000943c0, 0x814ea0, 0xc000202080)
C:/Go/src/net/rpc/server.go:465 +0x9e
created by github.com/DistributedClocks/GoVector/govec/vrpc.ServeRPCConn
C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/govec/vrpc/vrpc.go:38 +0xb6
This likely because under the hood go might be serializing the interface as a map.
Currently successfully instrumenting a directory using GoVector yields output looking like this:
$ GoVector -dir="example"
Source Built
Source Built
Wrappers Built
var conn net.PacketConn
var conn net.PacketConn
It'd be great to make the result more clear to help new users understand what's happing. It could look something like this:
$ GoVector -dir="example"
Reading example/main.go
Reading example/utils.go
examples/main.go:34 net.ReadFrom instrumented
examples/main.go:55 net.WriteTo instrumented
Overwriting original files
SUCCESS 2/2 detected network calls instrumented
if registerPublisher tries to register a nonce that already exists it should return an error to be dealt with in setupPubManagerTCP, otherwise the pubmanager will be shutdown for all potential new publishers.
The library is missing tests. This makes it difficult to know when things break and whether or not it really works. We need tests for each of the API calls (unit tests), and integration tests for combinations of API calls.
Unfortunately there appears to be a bug in the vector clock comparison implementation:
GoVector/govec/vclock/vclock.go
Line 138 in db949c8
This simple test will fail:
func TestCompareDifferentLengths(t *testing.T) {
n1 := New()
n2 := New()
n1.Set("a", 1)
n1.Set("b", 1)
n2.Set("b", 1)
n2.Set("c", 1)
n2.Set("d", 1)
if n1.Compare(n2, Descendant) {
failComparison(t, "Clocks are defined as Descendant: n1 = %s | n2 = %s", n1, n2)
}
}
The comparison function will detect that (B:1, C:1, D:1) is a descendant of (A:1, B1:), although they are concurrent.
I don't have a fix for this issue, but I'd like to warn future users of the package.
There does not seem to be any ongoing maintenance.
For now, we have created a fork of just the vclock
implementation: https://github.com/OpenFogStack/vclock
I can't import GoVector into the project and keep getting error where it is looking for
GoVector/govec/gopublisher.go:10:2: cannot find package "github.com/arcaneiceman/GoVector/broker" in any of: /usr/lib/go/src/pkg/github.com/arcaneiceman/GoVector/broker (from $GOROOT) /home/vm/gowork/src/github.com/arcaneiceman/GoVector/broker (from $GOPATH)
And then the same thing for gopublisher.go and other components. I added GoVector as git submodule to our project and did the import from the root of the project "project/GoVector/govec" where project
is root directory of our project. As far as I understand GOROOT needs to be set when you install go into a custom directory, but I haven't. GOPATH is working fine (so far) and is pointing to $HOME/gowork where the project is as well. Should GoVector be installed under $GOPATH as well for this to work? I was trying to make it self contained by having it as submodule.
Some info:
git submodule init
$GOPATH/src/project
$GOPATH/src/project/GoVector
$GOPATH/src/project/server
"project/GoVector/govec"
I would really appreciate some help. Might be something I am doing wrong. Thanks.
Currently the FileSub opens and closes the log file every time it writes a message. Change this to leave the file open and close it at the appropriate time.
This could be part of a deferred method call in the Broker to close all the connections to publishers and subscribers when the broker shuts down.
The broker uses the console to print verbose logging messages. Create a separate file to record these messages (separate from the vector clock log file intended for ShiViz).
Currently messages are printed to console.
Allow for a log file that is used for GoVec messages only (such as error and debug messages).
At the moment the broker code has a single web-socket. We need two, one for servicing publishers, and a second one for servicing subscribers.
We need to test the outputs written to Log files.
This would involve creating a mock library that can capture and test the outputs written to the Log File.
Something like this: https://medium.com/@matryer/5-simple-tips-and-tricks-for-writing-unit-tests-in-golang-619653f90742
Update LocalMessage and NetworkMessage to hold a byte representation of the vclock.
Update relevant portions of govec.go and gopublishers.go to handle this change.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.