paypal/gorealis

Version 1 of a Go library for interacting with the Aurora Scheduler

gorealis

Version 1 of Go library for interacting with Aurora Scheduler.

Version 2 of this library can be found here.

Aurora version compatibility

Please see .auroraversion to see the latest Aurora version against which this library has been tested.

Usage

• Getting started
• Using the sample client
• Leveraging the library

Projects using gorealis

• australis

Contributions

Contributions are always welcome. Please raise an issue to discuss a contribution before it is made.

gorealis's Issues

Add Support to create a Job to run container types

Right now the data structure are defines in thrift to accept containers. But there are no proper methods to modify the Taskconfig or Job struct.

Default to TBinaryProtocol

According to a new commit, there is benefit in using TBinaryProtocol apache/aurora@09b8e58

We should make this the default protocol for gorealis.

Specify field names when initializing structs

Currently, struct initialization of structs doesn't specify the field names to which the values are being assigned to. This method of struct initialization strongly depends on one being aware of the order of fields in the struct.

For example, in job.go, the initialization is done as aurora.MesosFetcherURI{value, &extract, &cache}.

Instead, for better readability and to prevent any code breaks, it might be better to initialize structs by also specifying the field names. Like below,

aurora.MesosFetcherURI{
    Value: value,
    Extract: &extract,
    Cache: &cache,
}

This will also ensure that the addition of new fields in MesosFetcherURI does not break existing code.

Nil error object returned from gorealis api for error scenarios

Nil error object returned from gorealis api error scenarios. this behavior is for both thermos and custom executor.

Purposefully set ram to 0 to generate error scenario ...

log for custom executor:
/usr/local/go/bin/go build -o /private/var/folders/jc/ljm2qnwx42l8f88vnxtt6k_82zcfdr/T/customCreatego /Users/kkrishna/work/golang/src/github.com/rdelval/gorealis/examples/client.go
Creating job
error obj:
response: Response({ResponseCode:INVALID_REQUEST Result_: ServerInfo:ServerInfo({ClusterName:aurora-cluster StatsUrlPrefix:}) Details:[ResponseDetail({Message:ram_mb must be greater than 0.0})]})Process finished with exit code 0

thermos logs:

/usr/local/go/bin/go build -o /private/var/folders/jc/ljm2qnwx42l8f88vnxtt6k_82zcfdr/T/thermosCreatego /Users/kkrishna/work/golang/src/github.com/rdelval/gorealis/examples/client.go
Creating job
error obj:
response: Response({ResponseCode:INVALID_REQUEST Result_: ServerInfo:ServerInfo({ClusterName:aurora-cluster StatsUrlPrefix:}) Details:[ResponseDetail({Message:ram_mb must be greater than 0.0})]})

Process finished with exit code 0

NewRealisClient method returning <nil> error when zookeeper leader not found

When Gorealis is not able to find Zookeeper leader (sample error response shown below), it does few retries. But if all the retries are unsuccessful, then the NewRealisClient method returns nil as error instead of any valid error.

ReestablishConn begin ....
error in LeaderFromZK:  No leader found

Logging levels

We need to add a logging library that allows for different levels.

Following Dave Cheney's blog on logging levels (https://dave.cheney.net/2015/11/05/lets-talk-about-logging), we will be sticking to two levels exclusively:

INFO and DEBUG

[Request] Add CI support for the library

Thanks for the awesome library! While I think we could add CI (unit and e2e test) and code coverage test for the repository to ensure quality.

Aggregate error handling and response checks at one place

Right now we handle a functions error from the function itself and go to response code check method for handle response validation. These are like two steps and common for every case and should be handled in one aggregate function.

For example

resp,err := somefunc(someargs)
return aggregatefunc(resp,err) should do 
-------
func aggregatefunc(response,error)response,error){
}

Monitor.Instances invoking os.exit(1)

https://github.com/rdelval/gorealis/blob/master/monitors.go#L82

return error instead of os.exit

Gorealis api error handling and resiliency

Gorealis client is for launching aurora tasks. An aurora task launch have multiple steps (thrift api) - launch followed by polling app count in active states. Typically, a complete launch task goes on for approx ~15 minutes depending on application bootstrap time. Currently, transport layer failures and aurora leadership changes are not handled in gorealis.

The goal is to handle following error scenarios:

• Transport layer failure, with retries.. this will require establishing thrift transport layer connection

• For aurora leadership change, api fails with http 5XX code. this will require finding leader from zookeeper and re-establishing thrift transport layer

Rename monitors from Noun Verb to Verb Noun

e.g. InstancesMonitor -> MonitorInstances

Not able to update CPU and Memory limits

Tried updating CPU and memory limits but aurora update jobs are failing to pick the latest values.

[feature] Allow config logging thrift response call separately

Now if we have realis.Debug() enabled entire job response will get print out, it makes log hard to read for both human and machine (a extreme long line)

// retry.go L152
r.logger.DebugPrintf("Aurora Thrift Call ended resp: %v clientErr: %v\n", resp, clientErr)

Some possible solutions

• add another client option realis.LogResp to enable/disable logging response regardless of log level
• add more logging level, debug, tracing .... (seems to be overkill)
• allow passing context into request, based on context, enable/disable log response on per request level instead of entire client

btw: I think it's a good idea to allow pass in context.Context as first parameter in all functions, this allow the caller to cancel the execution and pass in additional key value configs ...

GoRealis has uncaught exception inside GetInstanceIDs method

Gorealis commit ID being used : 94e7c878b4e54385b6fbbb7b0c1ce5a82865af2b

Find below the stack trace. Let me know if you need any further information.

realis-info: 2018/06/06 19:56:52 Client Error: HTTP Response code: 503
realis-info: 2018/06/06 19:56:52 Re-establishing Connection to Aurora
realis-info: 2018/06/06 19:56:52 Number of options applied to config: 4
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x467cf6]

goroutine 85075208 [running]:
bytes.(*Buffer).Write(0x0, 0xc42358c000, 0x141, 0x1000, 0x0, 0x0, 0x1)
/usr/local/go/src/bytes/buffer.go:136 +0x26
github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/vendor/git.apache.org/thrift.git/lib/go/thrift.(*THttpClient).Write(0xc450070d40, 0xc42358c000, 0x141, 0x1000, 0x732234, 0xc455d9ce60, 0x0)
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/vendor/git.apache.org/thrift.git/lib/go/thrift/http_client.go:212 +0x4d
bufio.(*Writer).Flush(0xc450070dc0, 0x72807e, 0xc455d9ce60)
/usr/local/go/src/bufio/bufio.go:566 +0x75
github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/vendor/git.apache.org/thrift.git/lib/go/thrift.(*TJSONProtocol).Flush(0xc427c4c080, 0x0, 0x0)
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/vendor/git.apache.org/thrift.git/lib/go/thrift/json_protocol.go:442 +0x36
github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/gen-go/apache/aurora.(*ReadOnlySchedulerClient).sendGetTasksWithoutConfigs(0xc455d9cf50, 0xc42827b9e0, 0xc434918bc8, 0x428e3e)
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/gen-go/apache/aurora/readonlyscheduler.go:371 +0x134
github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/gen-go/apache/aurora.(*ReadOnlySchedulerClient).GetTasksWithoutConfigs(0xc455d9cf50, 0xc42827b9e0, 0xc434918be0, 0xc434918bd8, 0xc44a427880)
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/gen-go/apache/aurora/readonlyscheduler.go:346 +0x39
github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis.(*realisClient).GetInstanceIds.func1(0xc400000008, 0xda03a0, 0xc4247f7b08)
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/realis.go:474 +0x3a
github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis.(*realisClient).thriftCallWithRetries.func1(0xc438f5c140, 0xc434918ce8, 0xc434918d58, 0xc434918e28)
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/retry.go:150 +0x6b
github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis.(*realisClient).thriftCallWithRetries(0xc438f5c140, 0xc434918e28, 0x2b, 0xc44700dbf0, 0x1)
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/retry.go:153 +0x222
github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis.(*realisClient).GetInstanceIds(0xc438f5c140, 0xc424b6a8d0, 0xc420017d40, 0xc434918f00, 0x4434e7, 0xc424b19756)
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/vendor/github.com/paypal/gorealis/realis.go:475 +0x198
github.paypal.com/PaaS-R/Genesis-Recon/aurora.retrieveTaskConfig(0x11be740, 0xc438f5c140, 0xc424b6a8d0, 0xd68688, 0x2, 0xc424b19740)
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/aurora/queryAurora.go:168 +0x62
github.paypal.com/PaaS-R/Genesis-Recon/aurora.GetAuroraData(0xc43e947720, 0x0, 0xc42035d920)
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/aurora/queryAurora.go:71 +0x150
main.checkForDataDiscrepency(0xc422443ee0, 0x12, 0xc422442680, 0x11, 0xc43e947720, 0xc442a8cd30, 0xc422b62000, 0xc422b620c0, 0xc42035d920, 0xc42035d9e0, ...)
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/main.go:584 +0x2f2c
created by main.beginOperation
/go/src/github.paypal.com/PaaS-R/Genesis-Recon/main.go:254 +0x2327

Rollback updates and cron style jobs

1. When an update is aborted, configuration doesn't rollback to previous state -- say that update was to increase instance count from 1 to 5 and it was aborted when instance count reached 4. It doesn't rollback to previous instance count of 1.

2. Also add support for scheduling cron-style jobs.

thanks.

Handle dedicated attributes misconfiguration more gracefully

There should be a check in gorealis whenever a dedicated attribute is added as a constraint so that it kicks back an error to the user if it's not configured properly.

The logic for setting up a dedicate attribute correctly can be found here:
http://aurora.apache.org/documentation/0.20.0/features/constraints/

Remove uncessary files from thrift dependency from V2

• Update .gitignore to not commit unnecessary files inside of thrift git repository.
• Point thrift git repository to the github address.
• Regenerate Thrift bindings with final 0.12.0 release.

gorealis config refactor

@rdelval ..Per our earlier conversation, this is the attempt..

The commit is in my fork: kkrishna@b10df06

Basically, with this change, to get a gorealis handle using, NewRealisClient api with variadic input options as follows:
r, err = realis.NewRealisClient(realis.ZKCluster(cluster), realis.BasicAuth(*username, *password), realis.ThriftJSON(), realis.TimeoutMS(15000))

r, err = realis.NewRealisClient(realis.SchedulerUrl(*url), realis.BasicAuth(*username, *password), realis.ThriftJSON(), realis.TimeoutMS(20000), realis.BackOff(defaultBackoff))

Pls. go through the code in my fork. Existing Default*APIs for client can be deprecated later. Will raise PR tomorrow after testing is complete. thanks.

Function visibility issues and incorrect function name in examples/jsonClient.go

In jsonClient.go, realis.NewDefaultConfig(...) shows an Unresolved reference error.
Assuming that this is the function that needs to be called, then we will need to capitalise the name from newDefaultConfig to NewDefaultConfig for inter-package visibility.

Also, in jsonClient.go#99 the call to realis.NewClient(...) shows an error as there is no function named NewClient in the realis package.
I looked through older versions of the code and found the function here.
Should one use NewRealisConfig(option ...ClientOption) instead?