skynetservices/skynet-archive

Skynet is a framework for distributed services in Go.

skynet-archive's Issues

missing ProtoMessage error

I try to install the skynet and get these errors:

github.com/4ad/doozer

c:\go\src\pkg\github.com\4ad\doozer\conn.go:184: cannot use &t.req (type *reques
t) as type proto.Message in function argument:
*request does not implement proto.Message (missing ProtoMessage method)
c:\go\src\pkg\github.com\4ad\doozer\conn.go:198: cannot use &r (type *response)
as type proto.Message in function argument:
*response does not implement proto.Message (missing ProtoMessage method)

I use the google protobuf 2.4.1. Does the skynet need the lower version of protobuf ?

Testing

Add unit testing to Skylib

Web UI

Show topology of the running services:

Datacenters
Hosts
Services by Service Version

At service version level, show metrics on response times.

Embed Graphite style dashboard for response time and request count monitoring, individually by service and aggregated by data center.

Add a FactoryImpl Generator

To keep my java friends happy, add a Generator that generates generators of Factories and Factory Implementations.

FactoryImplFactoryGenerator

We need a `release` branch.

Copied from issue#19:

I am using 'release' golang, not the latest, since I had a problem installing all libraries with the latest golang. But mango requires the latest golang, so I had to switch to an early version of mango.

We've had a good discussion on goinstall in the golang forum. goinstall will pull from a tag or branch called release if it exists. A release branch is better than a tag because it's easier to move later. I just helped @paulbellamy with that in mango, and it works great.

It sounds a bit complicated, but that's only because golang changes so often. It also means that you can worry less about breaking things on master, since most people will be pulling from release.

Testing

We need a plan for testing.

1. Generators
2. Generated apps

Generator testing is pretty straightforward. It either generates what we want or not.

Generated app testing is harder. an idea

1. we create a skynet application much like the one in examples - with full tests - and use it as the extraction source for the generators

I'm open to any others. Skynet won't be useful without a suite of good tests, so this is a priority.

Create generic Request and Response

Investigate creating dynamic request and response objects instead of currently strongly typed request and responses. Generic R & R would simplify a TON of code and allow all of the RPC stuff to be abstracted completely.

Benchmarks

Benchmark suite to check operations and discourage performance regressions.

Refactor Prototype Code

Much of this code is prototype quality. Refactor extensively. Fix pools, fix redundancy, fix inefficiencies, etc.

Events

Create named events based on lifecycle. Request start, complete, etc.

Create interface for event notification.

create implementation for event notification using a pub/sub (topic) message queue.

Timer on Services

Create an struct for service info that could be appended to an array of service infos. Then each service in the sky net chain can append metadata about the service request... processing time, node information, etc.

Introduce via 'example'. Describe skygen in wiki.

Brian, I think we should keep the README as simple as possible. We can skip the skygen/gb step by showing how to install/run GetWidgets or some other example. We can add tons of comments to that example for clarity.

Skygen can then be delineated on its own Wiki page, in much greater depth. Ok?

Python implementation

Since Ruby will get a gem, why not a library for Python?

Generator

Change the generator to be interactive. Instead of a bunch of ugly and hard to remember command line flags, let's use an interactive generator that builds up enough information to make a skynet mesh. This might be the highest bang for the buck right now.

Configurable Retry

Service should be marked as re-tryable. if service call fails, try again, up to limit or timeout.

Test Entire Suite

Create a detailed test script. Probably using a bash script. Start doozerd, start skynet instances. Test rpc, registration, deregistration, etc.

Routing methodologies

Right now, Skynet chooses a simple route randomly. May be very useful to choose routes other ways... perhaps by using a data center metric that gives affinity to closer nodes? More thought required on this.

Sky Command

Finish work on Sky command

Admin logic, etc.

Publish Skynet specification

Publish skynet specification so that we can run nodes in any language that implements the spec and supports the RPC protocol.

Code Cleanup

Code is generally ugly and needs refactoring, cleanup and full audit on error handling. Very few circumstances should allow a panic, most should handle any error condition gracefully from the end-user's perspective.

Abstract configuration (currently doozer)

Let's abstract out the doozer calls, so that we do not depend on doozer for testing.

zeroconf

support zeroconf for single subnet

Ruby Gem

When specification is more stable, create a ruby gem that enables a ruby node to participate in Skynet.

Reporting

Consolidate and display all of the fancy exported data that are under debug/expvar. Statistics are awesome - counts, averages, etc. Graphs are always good. Perhaps this is the start of a skynet dashboard.

License

Update source code - all code - to use a permissive open source license like MIT, that only requires attribution and has no restrictions for usage.

Create viewer to look at logs

Create web interface to review logs. Should be searchable in various ways, and should live stream to the screen perhaps via websockets.

Timeouts

Create a library to enable timeouts on all RPC calls, configurable at the client.

Doozer hardening

DzNS doesn't remove downed instances. Harden our doozer discovery implementation

Once DzNS has been setup ensure that we handle BootUri's properly and have environment variables to support them.

Ensure that we handle doozer connection failures, and find a new master if doozer can only write to master.

When doozer instances die and are restarted they loose all information, we need to make sure they reregister themselves.

Add more initiator examples

Add Twister, Web.Go, Resque and Stomp initiator examples

Spawner

We have a reaper, but no spawner. Need a spawner (maybe one on each physical host??) that has minimum threshold counts for number of each type of skynet nodes. If threshold is met, new instance of that type of node is spawned. Find a way to do this that doesn't require a ton of configuration.

Ruby Skylib implementation

Implement the functionality of skylib in a Ruby gem.

Service Admin

Start, Stop, Restart, Deregister services from sky command line interface.

Cache/Pool RPC connections

depending on eventual RPC implementation, it may be advantageous to cache or pool the connections to avoid the initial setup/teardown expense.

System Metrics Export

Export system metrics, load average, cpu, memory for collection by external process.

Problem with GetWidgets/router in examples

I have a problem compiled examples:

skynet/$ gb
(in examples/GetWidgets/initiators/mango) building cmd "mango"
(in examples/GetWidgets/router) building cmd "router"
router.go:15: can't find import: myStartup

By the way, I am using 'release' golang, not the latest, since I had a problem installing all libraries with the latest golang. But mango requires the latest golang, so I had to switch to an early version of mango. I'm curious how you manage to test everything. What versions are you using? If you try goinstall -clean -u on all dependencies, does it still work for you? (If that breaks for you, soon you'll be able to use my fork of mango. I learned that goinstall looks for a release tag if it exists.)

Change the transport of RPC calls

EDITED: Change all nodes to support both GOB and JSON-RPC

Metrics/Analytics

Each RPC service should export statistics on request timing, average, max, min, etc.

We should be able to see timing on each component of a composite service. If there are 10 RPC calls, then we should see timing on all ten as a piece of the composite.

Auto Port Assignment

When a new process starts, it should pick a port to listen on if none is specified.

Logging

Create interface for remote logging, and single implementation using mongodb capped collection. Interface should allow any implementation, first impl will be MongoDB capped collection.

Build Log viewer with searching into web UI.

Continuous Integration

Create CI server and publish CI stats.

Need any help?

Would you like any help on this project? If so, maybe we can use GitHub's issue-tracker, so that people don't work on the same thing at the same time.

Wiki

Build a Wiki page on how to get started. Build a wiki page on how to contribute.

Video Walkthrough / Demo

Create a screen cast of Skynet. Maybe do this after some of the UI issues are done - would be better impact with a visual dashboard showing nodes.

Interface to edit routes

Should be a pretty interface to edit routes, drag & drop -- should show a list of the available services.

Deployment

Create demon that runs on hosts and spawns instances of Skynet services. Should listen for commands on RPC interface on known port.

Skynet members die when doozer dies

Skynet members die when dozer dies ... this is probably not so indestructible.

Revisit Signal Implementation

Look at the signal implementation - see if anything can be added for convenience.

Pluggable configuration storage

Doozer is primary and makes most sense - but we should support some other options - via interface, selectable at generation time. Redis is a good choice because it supports push on changes. Others should be supported by implementing the interface.

Doozer HA

Change doozer code to connect to any member of the cluster, and reconnect on failure to another member.

More Examples

Create more examples - more initiator types, more chained services. Some async, etc.