won21kr/corteza-server-corredor

Corteza automation script runner

Corteza Corredor

Corredor is an automation script runner and bundler. It loads and processes scripts it has access and offers them to Corteza backend server on request.

It starts a gRPC server with 2 services: - Server scripts with list and exec procedures - Client scripts with list and bundle procedures

See protobuf service definition for details.

Corredor loads and monitors user scripts in the configured locations.

Main responsibilities

1. Watch and (re)load script dependencies (defined in standard package.json file)

2. Watch and bundle client scripts with webpack

3. Watch and reload server scripts

4. Listen for gRPC requests, provide list of client and server scripts, serve client script bundles and execute server side scripts

Roadmap

• pull git repositories (with scripts)

• automatically run unit tests for scripts

• secret and configuration management (key-value store)

Server scripts

These scripts are executed by Corredor directly or indirectly by the Corteza backend server. Scripts are not jailed or virtualized and run in the same context as the main Corredor app. This will probably be changed in the future to prevent scripts doing destructive actions such as stopping the Corredor server or similar.

Make sure your scripts are audited and accessing only limited data and making only changes they should. Last line of defense is (Docker) container, where the Corredor application is running.

Client scripts (WIP)

Client scripts are bundled with all their dependencies and 'Corteza webapp boot loader' with Webpack.

This bundles (each Corteza frontend app has one bundle) is loaded alongside the Corteza web application by the user’s browser.

Boot loader makes sure that all client scripts are properly registered in the event bus and executed in right order and in the right context

Make sure your scripts are audited and accessing only limited data and making only changes they should.

Running Corredor in production

docker run --rm -it \
  cortezaproject/corteza-server-corredor:latest

docker run --rm -it \
 --volume $(pwd)./custom-scripts:/corredor/usr:ro \
 cortezaproject/corteza-server-corredor:latest

Development & testing

Corredor uses YARN for package management. If you insist on using npm, make sure correct package versions are installed (yarn uses yarn.lock to keep track of exact versions used).

1. yarn to install dependencies

2. yarn serve:dev to start the development server (restarts when files change)

3. yarn test:unit to run all unit tests (add --watch watch files for changes and restart)

Note on Corredor’s package.json

You’ll see majority of development dependencies under dependencies key. This is intentional as we want to access certain packages in non-dev mode as well (bundling client scripts with webpack, Typescript compiling, running unit tests)

Running server with ts-node

We’re running corredor server with ts-node, source code is not transpiled with tsc nor bundled with rollup or webpack.

Reasons for these decisions:

1. We need to support server-side scripts These scripts are imported and executed in the same runtime as Corredor Scripts can import other libs and modules

2. This behaviour (importing, executing, loading external modules) should be same in production and development

3. scripts can have (multiple) package.json with defined dependencies that need to be loaded and included

4. one of the possible approaches is to bundle server-side scripts with tools like webpack or rollup. To date, we were not able to construct a feasible solution.

   • webpack issues: foliojs/fontkit#67.

   • rollup issues: issues with inclusion of external dependencies.

Environmental variables

Corredor read .env file if exists and merges that with environmental variables. See dotenv package for details.

All possible configuration is covered in src/config.ts where all environmental variables are read and processed.

Settings for both, Corredor and API server

Type	Default value	Description
CORREDOR_ADDR
string	depends on the environment	This setting is used by both, Corredor and API Server. For Corredor server: address where it’s gRPC server is listening on For API server: where can Corredor server be accessed. Note on default value and environment: By default (as configured in the source code), Corredor and API listen/connect-to localhost:50051. Docker images override these values with localhost:80 and corredor:80 so API can connect to Corredor in another container. If you change names of your Docker services, this value needs to be changed.
CORREDOR_LOG_ENABLED
boolean	corredor	This setting is used by both, Corredor and API Server. For Corredor service: where is service listening on (gRPC) For API server: where can Corredor service be accessed.
CORREDOR_LOG_LEVEL
string	info	Defaults to trace when in development environment, defines amount of log information outputed.

Corredor server

CORREDOR_LOG_PRETTY
boolean	false	Are events logged in one-line JSON or formatted to ease development?
CORREDOR_ENVIRONMENT
string	prod	Set environment we’re running in. This influences on other settings (see descriptions of other environmental variables). If CORREDOR_ENVIRONMENT is not found, Corredorr looks for CORREDOR_ENV and NODE_ENV variables (in that order). Recognised values are /^prod./ and /^dev./. In production: - server certificates are enabled (when not explicitly disabled by CORREDOR_SERVER_CERTIFICATES_ENABLED=false) - rich & pretty log output is disabled (when not explicitly enabled by CORREDOR_LOG_PRETTY=true) - log level is set to INFO (when not explicitly set with CORREDOR_LOG_LEVEL to trace, debug, info, warn, error)
CORREDOR_CORTEZA_PROTOBUF_PATH
string	rootdir + node_modules/corteza-protobuf	Corredor server is shipped with protobuffer definitions inside node_modules dir. You can change that if you need to test/alter proto buffers for development.
CORREDOR_SERVER_CERTIFICATES_ENABLED
boolean	true in production	Should Corredor gRPC require valid TLS certificates
CORREDOR_SERVER_CERTIFICATES_PATH
string	/certs	Path to Corredor server certificates
CORREDOR_SERVER_CERTIFICATES_CA
string	ca.crt	File name for the crtificate authority
CORREDOR_SERVER_CERTIFICATES_PRIVATE
string	private.key	File name for the private key
CORREDOR_SERVER_CERTIFICATES_PUBLIC
string	public.crt	File name for the public certificate
CORREDOR_SCRIPTS_AUTO_UPDATE_DEPENDENCIES
bool	true	Should corredor auto update dependencies every time scripts are refreshed?
CORREDOR_EXT_SEARCH_PATHS
string	./usr/*:./usr	List of paths (colon delimited) where Corredor should search for scripts
CORREDOR_SCRIPTS_SERVER_ENABLED
bool	true	Enable server scripts
CORREDOR_SCRIPTS_SERVER_WATCH
bool	true	Watch server scripts for changes and reload
CORREDOR_SCRIPTS_CLIENT_ENABLED
bool	true	Enable client scripts
CORREDOR_SCRIPTS_CLIENT_WATCH
bool	true	Watch client scripts for changes and reload
CORREDOR_EXEC_CSERVERS_API_HOST
string	public.crt	Where should Corredor connect to (on behalf of client scripts it is running)? This is used in combination with CORREDOR_EXEC_CSERVERS_API_BASEURL_TEMPLATE. If value is not found under CORREDOR_EXEC_CSERVERS_API_HOST, configuration tries to use DOMAIN, HOSTNAME and HOST. When starting up Corredor, you can see how Base URLs are set for each service.
CORREDOR_EXEC_CSERVERS_API_BASEURL_TEMPLATE
string	https://api.{host}/{service}	Template to assemble base url for system, compose and messaging REST API (CORREDOR_EXEC_CSERVERS_*\_API_BASEURL). Default value assumes corredor is running in monolith setup. If you are running subsystems in separated services, you can set this to https://api.{service}.{host}. When starting up Corredor, you can see how Base URLs are set for each service.
CORREDOR_EXEC_CTX_CORTEZA_SERVERS_SYSTEM_API_BASEURL
string	(assembled from baseurl template)	Set to a custom Base URL if you want to override value generated with CORREDOR_EXEC_CSERVERS_API_BASEURL_TEMPLATE.
CORREDOR_EXEC_CTX_CORTEZA_SERVERS_COMPOSE_API_BASEURL
string	(assembled from baseurl template)	Set to a custom Base URL if you want to override value generated with CORREDOR_EXEC_CSERVERS_API_BASEURL_TEMPLATE.
CORREDOR_EXEC_CTX_CORTEZA_SERVERS_MESSAGING_API_BASEURL
string	(assembled from baseurl template)	Set to a custom Base URL if you want to override value generated with CORREDOR_EXEC_CSERVERS_API_BASEURL_TEMPLATE.
CORREDOR_EXEC_CTX_FRONTEND_BASEURL
string	(can use PROVISION_SETTINGS_AUTH_FRONTEND_URL_BASE or falls back to DOMAIN, HOSTNAME and HOST variables)	Provide frontend base URL info for scripts (accessible through context).

Corteza API

CORREDOR_ENABLED
bool	true	This is a setting for API server, will Corredor be used for server automation?
CORREDOR_MAX_BACKOFF_DELAY
duration	1m	Connection timeout (from API server to Corredor) CORREDOR_DEFAULT_EXEC_TIMEOUT
duration	1m	Timeout for server script execution CORREDOR_LIST_REFRESH
duration	30s	Script list refresh frequency CORREDOR_LIST_TIMEOUT
duration	2s	Timeout when fetching list of client or server scripts CORREDOR_RUN_AS_ENABLED
bool	true	Allow running server scripts as another user
CORREDOR_CLIENT_CERTIFICATES_ENABLED
boolean	true in production	Should Corredor gRPC client connect with valid TLS certificates
CORREDOR_CLIENT_CERTIFICATES_PATH
string	/certs/server/server	Path to Corredor client certificates
CORREDOR_CLIENT_CERTIFICATES_CA
string	ca.crt	File name for the crtificate authority
CORREDOR_CLIENT_CERTIFICATES_PRIVATE
string	private.key	File name for the private key
CORREDOR_CLIENT_CERTIFICATES_PUBLIC
string	public.crt	File name for the public certificate
CORREDOR_CLIENT_CERTIFICATES_SERVER_NAME
string		Server name to use on connection

BloomRPC

RPC server can be manually tested with any gRPC client. If you do not have your favorite, we recommend BloomRPC.

Testing with BloomRPC client & secure server:

1. Click on TSL button on top

2. Add root certificate (ca.crt)

3. Private Key (private.key)

4. Cert Chain (public.crt)

Development plan (unordered)

• Bundling and serving Vue.js components

• Bundling and serving SCSS and binary content

• Updating files via gRPC service

• Support for remote (git repository) location for user scripts

• Support for basic git operations

• Automatically running user script unit tests before loading scripts

• Support TypeScript for user scripts