gojek/proctor

A Developer-Friendly Automation Orchestrator

Proctor

Description

Proctor is a developer friendly automation orchestrator. It helps everyone use automation and contribute to it

Dev environment setup

• Install and setup golang
• Clone the repository
• Run make build

proctord

• proctord is the heart of the automation orchestrator
• It is a web service that handles management and execution of procs

Dev environment setup

• Ensure local postgres server is up and running
• Ensure local redis server is up and running
• Install kubectl
• Configure kubectl to point to desired kubernetes cluster. For setting up kubernetes cluster locally, refer here
• Run a kubectl proxy server on your local machine
• Configure proctord
• Setup & Run database migrations by running this command make db.setup from the repo directory
• Start service by make start-server
• Run curl {host-address:port}/ping for health-check of service

proctord configuration

• Copy .env.sample into .env file
• Please refer meaning of proctord configuration here
• Modify configuration for dev setup in .env file
• Export environment variables configured in .env file by running source .env
• proctor server gets configuration from environment variables.

proctord configuration explanation

• PROCTOR_APP_PORT is port on which service will run
• PROCTOR_LOG_LEVEL defines log levels of service. Available options are: debug,info,warn,error,fatal,panic
• PROCTOR_REDIS_ADDRESS is hostname and port of redis store for jobs configuration and metadata
• PROCTOR_REDIS_MAX_ACTIVE_CONNECTIONS defines maximum active connections to redis. Maximum idle connections is half of this config
• PROCTOR_LOGS_STREAM_READ_BUFFER_SIZE and PROCTOR_LOGS_STREAM_WRITE_BUFFER_SIZE is the buffer size for websocket connection while streaming logs
• PROCTOR_KUBE_CONFIG needs to be set only if service is running outside a kubernetes cluster
  • If unset, service will execute jobs in the same kubernetes cluster where it is run
  • When set to "out-of-cluster", service will fetch kube config based on current-context from .kube/config file in home directory
• If a job doesn't reach completion, it is terminated after PROCTOR_KUBE_JOB_ACTIVE_DEADLINE_SECONDS
• PROCTOR_KUBE_JOB_RETRIES is the number of retries for a kubernetes job (on failure)
• PROCTOR_DEFAULT_NAMESPACE is the namespace under which jobs will be run in kubernetes cluster. By default, K8s has namespace "default". If you set another value, please create namespace in K8s before deploying proctord
• PROCTOR_KUBE_CLUSTER_HOST_NAME is address/ip address to api-server of kube cluster. It is used for fetching logs of a pod using https
• PROCTOR_KUBE_CA_CERT_ENCODED is the CA cert file encoded in base64. This is used for establishing authority while talking to kubernetes api-server on a public https call
• PROCTOR_KUBE_BASIC_AUTH_ENCODED is the base64 encoded authentication of kubernetes. Enocde username:password to base64 and set this config.
• Before streaming logs of jobs, PROCTOR_KUBE_POD_LIST_WAIT_TIME is the time to wait until jobs and pods are in active/successful/failed state
• PROCTOR_POSTGRES_USER, PROCTOR_POSTGRES_PASSWORD, PROCTOR_POSTGRES_HOST and PROCTOR_POSTGRES_PORT is the username and password to the postgres database you wish to connect to
• Set PROCTOR_POSTGRES_DATABASE to proctord_development for development purpose
• Create database PROCTOR_POSTGRES_DATABASE
• PROCTOR_POSTGRES_MAX_CONNECTIONS defines maximum open and idle connections to postgres
• PROCTOR_POSTGRES_CONNECTIONS_MAX_LIFETIME is the lifetime of a connection in minutes
• PROCTOR_NEW_RELIC_APP_NAME and PROCTOR_NEW_RELIC_LICENCE_KEY are used to send profiling details to newrelic. Provide dummy values if you don't want profiling
• PROCTOR_MIN_CLIENT_VERSION is minimum client version allowed to communicate with proctord
• PROCTOR_SCHEDULED_JOBS_FETCH_INTERVAL_IN_MINS is the interval at which the scheduler fetches updated jobs from database
• PROCTOR_MAIL_USERNAME, PROCTOR_MAIL_PASSWORD, PROCTOR_MAIL_SERVER_HOST, PROCTOR_MAIL_SERVER_PORT are the creds required to send notification to users on scheduled jobs execution
• PROCTOR_JOB_POD_ANNOTATIONS is used to set any kubernetes pod specific annotations.
• PROCTOR_SENTRY_DSN is used to set sentry DSN.

proctor's Issues

Add Proctor Api Documentation

Add readme for setting up automation orchestrator

As a user,
I would like to see how to stitch all components of Proctor together.

Currently, dev setup for proctor CLI and proctord exists.

What already exists:

• Setup proctord with a secrets store, procs engine and audit logging store
• Setup proctor CLI to talk to proctord

Acceptance criteria:

• Setup CI for packaging procs inside images and push to registry
• Setup CI for procs metadata to be populated in proctord

Remove proc command from proctor CLI

As a user,
I don't want to type the redundant word proc everytime I list/describe/execute procs.

Don't display `Constants` when asked for help on a proc

As a user,
I mostly see help to run a proc. Since I don't have to provide constants to run a proc, it deteriorates my experience.

An admin can fetch the required constants to submit secrets for a proc from the procs metadata file. So, it makes sense to remove display of Constants from the CLI.

Notification on Proc completion

As a user,
I want to be notified when the proc I executed completes.

So that I don't have to pay attention to the progress of procs in execution, and can focus on something else.

Once a proc reaches completion, proctor should notify me about that, and let me know if the proc completed successfully or failed.

Out of scope:
Multiple channels of notification.
For now, proctor will only accept an api during executing a proc. It will notify on that api once the proc reaches completion.
Implement multiple channels for notification later.

Add meta fields author, contributor and organization to proctor scripts

Mandatory variable check without executing Procs

Currently, Proctor CLI will do variable check by executing procs on the executor.
Need to add a feature on CLI to check Mandatory Variable without contacting Proctor Service.

Notify users on new release of Proctor CLI

As an admin,
I want to notify CLI users when a newer version of proctor is released.

So that,
Users can stay updated with the latest features.

When a newer version of proctor (compared to the installed version) is released, Users should see a notification on every command run from CLI. (Example: gcloud/aws CLIs)

Run procs in specified environment

As an admin,
I want users to be able run procs in the environment they specify.

So that,
Connectivity issues won't arise while running procs.

Currently procs are run in a single predefined environment. Hence, users need to ensure connectivity to separate environments. This feature will increase the capabilities of procs and reduce the headache of maintaining connectivity with multiple environments.

Validate .proctor.yaml configuration

0. Validate $HOME/.proctor/proctor.yml file exists, if not give instructions on user about creating the yaml files
1. Validation .proctor.yml file's presence and check for mandatory parameters like PROCTOR_HOST, EMAIL_ID and ACCESS_TOKEN. Give warning if parameters missing.
2. If received 401 from server, check for presence of EMAIL_ID and ACCESS_TOKEN and ask for setting the same
   2.1 If received 401, and EMAIL_ID and ACCESS_TOKEN present, give user warning on invalid ACCESS_TOKEN and regenerate one
3. If unable to reach host, print out error message for reachability and checking internet/vpn connection

Persist fetching the procs execution status job

As an admin,
I want to persist the procs execution status job for a worker to pick it up.

Currently, it is being run asynchronously using a goroutine. In that case, if there's a deployment of proctord, the proc execution status won't be updated in the database.

Add Makefile for proctor-cli

• Add Makefile for proctor-cli

Remove the redundant `execute` command for proc execution

As an admin,
I don't want users to type execute everytime they want to execute a proc.

Current scenario:

proctor execute create-some-infra-one ARG_ONE=a ARG_TWO=b
proctor execute create-some-infra-two ARG_THREE=c ARG_FOUR=d

Better scenario:

proctor create-some-infra-one ARG_ONE=a ARG_TWO=b
proctor create-some-infra-two ARG_THREE=c ARG_FOUR=d

This will also help move towards resolving #32

Incorrect error message when running jobs without pemissions

Proctor shows the following error message when I try to run procs that I don't have permissions for.
Error submitting proc for execution

It should show a better error message so the user knows why the proc failed.

Proctor CLI exits with non-zero status on proc failure

As an admin,
I want the proctor CLI to exit with non-zero status when a proc fails.

So that,
They can use proctor CLI inside their scripts and have proper feedback on it's success/failure.

Technical details:
Post logs streaming is completed, poll proctor daemon for the status of proc until it's successful/failed.

Encrypt secrets while storing them

As an admin,
I want to encrypt secrets while storing them using proctord.

So that,
If secrets get leaked accidentally, they aren't of any use without the encryption key.

Not able to delete procs from proctor.

I have removed one job from my proctor job repo and I have updated the metadata without the removed job. But still I can able to see the job in the proctor proc list

Refactor auditing of proctor jobs

Currently,
Golang's context is used to pass auditing metadata. This is a bad practice.

Instead,
We could have a builder object and pass it for auditing at the end. That will be more cleaner.

Display authorized groups when describing a proc

As a user,
I want to see authorized groups for a proc when seeing the help for it.

Add uniqueness constraint on job names submitted for execution

As an admin,
I want to add uniqueness constraint on column job names submitted for execution.
Because:

• As of now, proctord has only one engine to run jobs. So there shouldn't be any conflicting names
• As number of records grow, the query to fetch jobs by execution names will become slower

Alternative methods to specify input parameters for procs

When input parameters for a proc are large, it sometimes becomes difficult to verify the input params one last time before executing a proc. Does it make sense to add a feature to the CLI to read input params from a JSON file?

[Bug] Fix job execution status updation logic

In kubeclient, on job failure, it doesn't return the status immediately and keeps on watching for more events

• The event watching channel isn't being closed and can lead to file descriptor leaks.
• Add an updated at column in database when updating a record.

Schedule Procs

As a user,
I want to schedule procs to run at a defined time and date.

So that I don't have to manually execute the proc at that time. proctor gets executes the proc.

Out of scope:
Only schedule a proc to run once. For executing procs in a recurring schedule, implement later.

Provide free-flowing args while executing a proc

Currently, for providing any args to a proc, I need to pass them as K-V pairs.
Example:

proctor execute grep-knife-node-list NODE_NAME=proctor-postgres

Would it make sense to pass free-flowing args while executing a proc?

Example:

proctor execute grep-knife-node-list proctor-postgres proctor-redis

Details:
proctord can provide all the free flowing args as a default(convention) env var to a proc
The proc will have logic to parse the args from default env.

Fix PROCTOR_URL behaviour

The PROCTOR_URL name of the config variable implies that it must be a URL. But in the code, it is used as the hostname. This is confusing.

Example lines:
https://github.com/gojektech/proctor/blob/master/daemon/client.go#L42
https://github.com/gojektech/proctor/blob/master/daemon/client.go#L64

Enable auto-completion in bash for basic commands

We can have auto-completion for describe/execute/list commands.

• Out of scope:

auto-completion for procs and their args.

Add Email-Id and Access-Token Headers for authentication

As an admin,
I want users to send Email-Id and Access token when interacting with proctorD from proctor-CLI.

So that, I can audit users' name on execution of procs, submission of secrets, etc.

Intuitive help for procs

As a user,
I want procs help to be more intuitive.

Currently, If I fetch all procs using proctor list, and I need help for run-sample proc, I need to type command: proctor describe run-sample. That's unintuitive.

Instead, proctor help run-sample or proctor run-sample --help would be more intuitive for displaying help.

Incorrect error messages on proctor CLI on proctord failure

When an authorized user executes a proc,
If proctord responds with a non-successful response,
The user is shown an error: Error executing proc. Please check configuration and network connectivity
Instead the error message should read: Proc execution failed. Please consult admin.

Stream logs from api-server directly

• Currently kube proxy is used for streaming logs
• This will reduce one point of failure for proctor

Incorrect doc for proctord config

In code it expects: PROCTOR_KUBE_BASIC_AUTH_ENCODED as key of kube basic auth.

In doc it mentions: PROCTOR_KUBE_BASIC_AUTH

Notice to upgrade proctor CLI when out of date

As an admin,
I want to notify users when new version is available whenever they run proctor version

Example:

$terraform version
Terraform v0.11.7

Your version of Terraform is out of date! The latest version
is 0.11.10. You can update by downloading from www.terraform.io/downloads.html

Timeouts on CLI when connecting to proctord

The CLI currently has infinite timeout when connecting to proctord. If there's connectivity issue, the user won't know. Instead have a sane timeout to fail fast and give user relevant feedback.

Have proctor CLI update itself

As an admin,

Post being notified from the CLI on a new release of Proctor CLI, users can update proctor from the CLI itself. Example: proctor update <version(optional)>

Notify CLI users when newer version of proctor is released

As an admin,
I want to notify CLI users when a newer version of proctor is released.

So that,
Users can stay updated with the latest features.

When a newer version of proctor (compared to the installed version) is released, Users should see a notification on every command run from CLI. (Example: gcloud/aws CLIs)

List procs in alphabetical order

When showing the list of procs, for enhancing UX. It would be nice to have the list of the procs to be shown in an alphabetical order.

Procs are rerun on failure

As an admin,
I want to configure the number of retries for procs on failures.

So that,
Users contributing to procs can have control if they haven't handled immutability, state, etc. A proc will be run once under all circumstances. On failure, it can be configured to not rerun.

Currently, it's retried 6 times on failure by default.

Support for Multi Proctor profiles

Just like AWS CLI, it will be better if procs support multi-profile, so we can use proctor for more than one proctor service.

Currently, if we want to use proctor to other proctor services, we will need to modify proctor config file.

Load config only once per run

• Move InitConfig to main.go
• Don't make config as Singleton
• Clean-up the existing test for configs

Track job execution status

As an admin,
I want to track which jobs submitted for execution succeeded and failed.

Every job which is successfully submitted to proctor, will be running as a script. The job success/failed will depend on exit code of the script. If exit code is 0, the job ran successfully. Else, failed.

Integrate newrelic instrumentation with proctor

I want to instrument the code, enable application-level metrics for proctord.

So that,
In the case of buggy behaviour, I can correctly diagnose the root cause of the problem.

Throws error on virgin proctor config command invocation

Should automatically create whatever directory & write configuration files it needs. I do not understand why does it throw this error.

Getting the issue while installing proctor.

github.com/spf13/cobra
gojek/src/github.com/spf13/cobra/command.go:31:25: undefined: pflag.ParseErrorsWhitelist
github.com/googleapis/gnostic/extensions
gojek/src/github.com/googleapis/gnostic/extensions/extension.pb.go:57:29: undefined: proto.InternalMessageInfo
gojek/src/github.com/googleapis/gnostic/extensions/extension.pb.go:121:45: undefined: proto.InternalMessageInfo
gojek/src/github.com/googleapis/gnostic/extensions/extension.pb.go:180:46: undefined: proto.InternalMessageInfo
gojek/src/github.com/googleapis/gnostic/extensions/extension.pb.go:237:29: undefined: proto.InternalMessageInfo
github.com/gojektech/proctor/proctord/storage
gojek/src/github.com/gojektech/proctor/proctord/storage/store.go:73:33: multiple-value uuid.NewV4() in single-value context

Getting status of a proc submitted to proctor service

POST /jobs/execute returns a job-id to the user. But there is no way to get status of a job submitted to service.

Let's expose an API to fetch the status of a job.

Add init command that bootstrap proctor config

Add proctor init command for first time user to generate proctor config file based on parameters.

Fix discrepancy between jobs/procs/automation

As an admin,
I want to clean-up inter-changeable usage of the keywords jobs, procs, scripts and automation inside Proctor. Across Proctor, only 1 keyword should be used for the automation run.

So that,
A user isn't confused. Usage of only one keyword will help achieve clarity in the mind of users using Proctor.

Namespace config of CLI under proctor

As an admin,
I don't want existing config of users to be accidentally used when running proctor. Currently, CLI has config with keys EMAIL_ID and ACCESS_TOKEN. Since these are common keys, user might have them set as ENV vars.

When proctor runs, it might pick preset config without user's knowledge.

In order to avoid such behavior, we can namespace config under PROCTOR_. Example: PROCTOR_EMAIL_ID and PROCTOR_ACCESS_TOKEN

Retry logs streaming from CLI on failure

Currently, CLI (client) doesn't retry on failure while streaming logs. This removes the ability to deploy proctord at desirable hour.

Move dependency management to go modules

As of go 1.11, modules is the official dependency management tool in golang.

It makes sense to move proctor to use modules. Changes that might be required:

1. Change the directory structure from GOPATH to outside GOPATH
2. Have one dependency manager for both proctor CLI and proctord (currently we have 2 glide.yaml files)
3. Have one binary for CLI and proctord

Test proctor client config setup command

As an admin,
I want to test logic written inside config setup.

So that,
On modifying it in future, existing functionality doesn't break.