:shipit: Hal Deployment Platform - Web UI/API
License: MIT License
Hal is a configuration and deployment management platform for private datacenters, AWS, and more.
It contains a Web UI/API and agent for running jobs and long-running tasks. This repository is for the Web UI/API.
Table of Contents:
TBD
TBD. We'll build out this section once Hal 3.0 is closer to release.
TBD. We'll build out this section once Hal 3.0 is closer to release.
mcp-core to mcp-commonpanthor-pluginsmcp-corp-account and figure out what to do about login (for now, oauth2 later)Source Code
Uncompiled code pulled directly from VCS repository.
Build
Build artifact or dist. This is the source code after build or compilation commands have been run.
Environment
A datacenter tier. Typically, application code is promoted and tested in multiple environments before it reaches production.
Group
A host server, region or other container for an application to be deployed to.
Groups have a type (rsync, generic, elastic beanstalk, s3, etc)
Groups have a name (server hostname, region, etc)Example: Group "CodeDeploy" for "us-east-1". Targets with specific CD details can be attached to this Group.
Target
A location or service configuration to deploy a build to. A target consists of a group and additional details. An application can deploy to multiple targets on a host (multiple directories on a server, multiple apps in an AWS region, etc).
Targets have parameters for the group they belong to (e.g. codedeploy app name and deployment config for CodeDeploy groups)
Release
Resource representing a deployment action. No release artifact is stored, this is entirely metadata about the deployment process.
Deploy
Deployment must never be used in documentation or UI. Deploy is an action.
Example: "Can you generate a build so it can be deployed later tonight at the scheduled time?"
correct!
- Generate a build
- Create build
- Run build
- Deploy this build to the test environment.
- Build is in process
- Build is running
wrong!
- Build a build
- Build the application
- The application is building
correct!
- Deploy a release
- Deploy this release to the test environment.
- Deploy release
- Release is deploying
wrong!
- Create a deployment
- Run deployment process
- Push the deploy
- Deployment is running
Push
Deployment
Deployment is a term used to describe the overall system, but is not used within the system.
E.g. "Deployment system". "Hal handles deployments for my application"
Hal should be able to pull code from TFS / VS.com and also deploy releases through them.
API docs: https://www.visualstudio.com/en-us/docs/integrate/api/overview
Hal should have a way for admins to change the configuration of Hal more easily (without requiring redeployment with new static configuration.
.hal.yml)Should be MIT license
When a user selects to have a release run automatically after a successful build, the status of these child job processes are not automatically updated on the build page.
Previous:
ql/hal - QL\Hal\*ql/hal-agent - QL\Hal\Agent\*ql/hal-core - QL\Hal\Core\*New:
hal/hal - Hal\UI\*hal/hal-agent - Hal\Agent\*hal/hal-core - Hal\Core\*/ (Dashboard)/signin/signout/help/help/deployment-scripting/help/application-setup/help/github-permissions (removed)/styleguide/styleguide/icons/latest-release/queue/queue-history/queue-history/date/$date/application/$app/builds/application/$app/builds/page/$page/builds/$build/applications/$app/build/application/$app/pushes/application/$app/pushes/page/$page/pushes/$push/applications/$app/push/applications/$app/deployments/$deployment/pushes/applications/$app/deployments/$deployment/pushes/page/$page/servers/servers/add/servers/$server/servers/$server/edit/servers/$server/remove/organizations/organizations/add/organizations/$org/organizations/$org/edit/organizations/$org/remove/environment/environment/add/environment/$env/environment/$env/edit/environment/$env/remove/users/users/$user/users/$user/disable/users/$user/enable/settings/settings/favorite-applications/$app/settings/favorite-applications/$app/delete/users/$user/tokens/users/$user/tokens/$token/revoke/applications/applications/add/applications/$app/applications/$app/edit/applications/$app/remove/applications/$app/dashboard/applications/$app/environments/$env/applications/$app/deployments/applications/$app/deployments/add/applications/$app/deployments/$target/applications/$app/deployments/$target/edit/applications/$app/deployments/$target/remove/applications/$app/environments/$env/pools/applications/$app/environments/$env/pools/add/deployment-views/$view/deployment-views/$view/edit/deployment-views/$view/remove/deployment-views/$view/pools/add/deployment-views/$view/pools/$pool/edit/deployment-views/$view/pools/$pool/remove/deployment-views/$view/pools/$pool/add/deployment-views/$view/pools/$pool/remove/$target/applications/$app/encrypted-properties/applications/$app/encrypted-properties/add/applications/$app/encrypted-properties/$encrypted/applications/$app/encrypted-properties/$encrypted/remove/admin/admin/audit-logs/admin/audit-logs/page/$page/admin/status/admin/status/$agent/superadmin/caches/superadmin/users/superadmin/message/admin/credentials/admin/credentials/$credential/admin/credentials/$credential/add/admin/credentials/$credential/remove/admin/permissions/admin/permissions/$perm/remove/admin/permissions/$user/add/admin/permissions/$user/refresh/admin/permissions/deployment/admin/permissions/deployment/$perm/remove/api/internal/github-users/$user/repositories (removed)/api/internal/applications/$app/environments/$env/status/api/internal/settings/favorite-applications/$app/api/applications/$app/build/docs/api/api/api/environments/api/environments/$env/api/servers/api/servers/page/$page/api/servers/$server/api/organizations/api/organizations/$organization/api/users/api/users/page/$page/api/users/$user/api/targets/$target/api/targets/$target/current-release/api/targets/$target/pushes/api/targets/$target/pushes/page/$page/api/builds/api/builds/$build/api/builds/$build/events/api/pushes/api/pushes/$push/api/pushes/$push/events/api/job-events/$event/api/queue/api/queue-refresh/$jobs/api/applications/api/applications/$app/api/applications/$app/deployments/api/applications/$app/builds/api/applications/$app/builds/page/$page/api/applications/$app/pushes/api/applications/$app/pushes/page/$pageFurther refinement of #8 (Refactor domain model)
Add "Source"
- Contains details of where source code is located.
- No credentials, tokens, endpoints, or other source details should be hardcoded in hal configuration.
- Type (ex: github, gitlab, git, tfs, etc)
- Endpoint (ex: https://api.github.com)
- Credential
- Add new credential type
Remove "Group"
- Targets do not need to be attached to a "server" or "group" and can exist on their own (still requires environment)
- Add "Target Template" (need to workshop the name a bit)
- This contains the same details as targets, but all information is optional.
- This allow us to create a "target template" for "AWS us-east-1" for example, and all targets that use that template (or inherit from it) will get that information automatically.
- A template is not required to create a Target.
Examples
In the following examples, the "rsync" target type needs the following params:
(server name, path)
- Template A:
- Type: rsync
- Server name: localhost
- Target B (inherits from template):
- Template: TemplateA
- Path: /var/www/website
- Target B (no template):
- Type: rsync
- Server name: example.com
- Path: /opt/myapp
Need to update the schema for this file to match terms from #9
Also the file should be changed to .hal.yml
hal-core, hal-agent and hal-ui should all have config extensions for defining their required config parameters.
The doctrine entities changed significantly.
Formerly known as Kraken.
This system will be a method to deploy "configuration" instead of "code".
This will require a significant amount of work because a lot of settings are duplicated in the database / domain model (apps, environments, etc) and will need to be removed. We'll also need to re-architect how the configuration cluster operates and is maintained.
Ideally this system would be flexible enough to support consul, etcd, zookeeper, etc.
Kraken will become the "runtime encrypted configuration" component of Hal.
If a server command is found in this form: file:/file/must/exist/here.
Hal should move the file to the remote server (hash and randomize the scriptname), run it, then delete it. See ansible for reference.
This will allow prepush scripts to run before deploying to the server (Since the script will not exist yet).
The current rate limiting middleware for the API is naive and very basic. Though it's really only designed to prevent errant scripts or bad code from crushing the server resources, not for handling general traffic spikes, request priorities and shedding load. While we do rate limit per user x per endpoint, we can only customize this per endpoint, not per user.
We should implement the token bucket algorithm described here by stripe engineering:
https://stripe.com/blog/rate-limiters
Also see code examples here:
https://gist.github.com/ptarjan/e38f45f2dfe601419ca3af937fff574d
Kraken is the system for providing runtime configuration for applications that use Hal. It deploys configuration instead of code.
Will be re-added at a later time.
To allow more targeted and user-friendly messages before making destructive actions. This allows
a template to better curate its content for the entity being removed (or maybe use different templates).
See the following psr-7 middleware for an example
class DeleteConfirmationMiddleware implements MiddlewareInterface
{
use EntityLoaderTrait;
use HTTPMessageTrait;
/**
* @var TemplateInterface
*/
private $template;
public function __construct(TemplateInterface $template)
{
$this->template = $template;
}
/**
* @param ServerRequestInterface $request
* @param ResponseInterface $response
* @param MiddlewareInterface|callable $next
*
* @return ResponseInterface
*/
public function __invoke(ServerRequestInterface $request, ResponseInterface $response, callable $next)
{
if ($request->getMethod() === 'POST') {
return $next($request, $response);
}
$entities = [];
foreach ($this->entityMappings() as $uriKey => $type) {
try {
$entity = $this->getEntity($request, $uriKey, $type);
$entities[$uriKey] = $entity;
} catch (InvalidArgumentException $e) {}
}
$rendered = $this->template->render([
'delete' => $entities
]);
return $this->withBody($response, $rendered);
}
}
Currently it is not possible to use different credentials per host without redeploying hal-agent.
This should be managed within the db (encrypted, obviously) and allow multiple types (pw, filesystem keyfile, provided keyfile). Deployments would then be assigned a credential and they would use that when deploying.
Server Commands should be able to use a different credential from deployment.
AWS deployments currently use this system, but rsync-based deployments are more complex, as we need to be able to connect using native ssh with php, but also command line rsync.
Perhaps we need to figure out a good way to pass db-based credentials to the cli rsync.
PHP Sec lib has finally reached stable 2.0 which adds a better api with namespaced classes and improved error handling (e.g. not using trigger_error).
This includes all dependencies.
Probably includes the following:
Configuration secrets should be passed in the environment instead of stored in the DI container.
Symfony has supported ENV-based runtime parameters since 3.2 - http://symfony.com/blog/new-in-symfony-3-2-runtime-environment-variables
A dotenv component was introduced in 3.3 - http://symfony.com/blog/new-in-symfony-3-3-dotenv-component.
While it is best for any infrastructure of hal to provide these values in the env, we can also store these values in env.yml files for environments such as dev.
I would like the hal ecosystem to be more heterogeneous and less PHP-focused. In support of that, maybe the easiest way to do that is an elixir and phoenix based API. I like those technologies.
Since the API is currently (mostly) readonly, it would be a good time to consider writing the API in another language.
This project would be a separate repository (hal-api) and use elixir 1,4, phoenix 1.3, and ecto for database interactions.
The doctrine entities changed significantly.
There is a pattern that large systems use to handle data migration with zero time - Typically this involves a multi-step migration process:
Etsy and Stripe have written about their experiences with this:
Hal currently uses the doctrine ORM for database management. We need to investigate if using such a pattern is even possible with it. Ideally migrations can happen safely and reliably enough that they can occur while using Continuous Deployment. Not only would this improve our own instance, but also provide a better experience to people using the open source version of Hal.
Images should be stored in a docker registry.
Also remove some of the util console commands like redeploying the docker images. We dont want to be managing dockerfiles on the filesystem of builders.
Currently each agent only supports a single build server. This should be changed to support a list of build servers, and the agent should randomly pick one when a build (or deployment) is initiated.
Using self-discovery for identifying build servers would be a huge enhancement, but should not be part of Hal 3.0 as we do not have the time to do proper R&D on it.
Currently the agent requires cron jobs for processing builds and releases.
"Pending" jobs are read from the database by the agent it will immediately start all pending jobs. We need a better way to process jobs, a method that uses queues and allows throttling or capacity management.
Eventually I'd like the agent to not need direct db access so that is something to consider.
Allow Hal to use OAuth2 based logins such as github, ping, google, etc.
Note: LDAP should still be an option.
We should have a terraform and docker configurations for getting a working hal environment up and running as quickly as possible.
Terraform enables you to safely and predictably create, change, and improve production infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
Show all permissions for
When the permission system was rebuilt, permission listings were removed.
These should be added back to a dedicated page, as computing all permissions can take some time. This should also reach out to github and be able to determine if a user has permission (github org), but has not logged into to hal yet.
Example:
test_propertyinstead oftest-property
Some values defined as unique in the doctrine yaml files are not created with the unique constraint by the phinx migrations.
All fields that have unique constraint defined in orm.yml should be generated with constraints in phinx migrations
The following constraints are missing
There are some things that are not set as unique that maybe should be -- Users email, Organization names, AWS key?
As we run all insecure command such as build an deploy commands in docker containers, we need to ensure our docker environment is secure and safe from injection, jailbreaking, or other nefarious commands.
Can the following example break hal? Then this issue isn't complete.
https://github.com/amoffat/dockerbomb
Resources:
Currently Hal forces users to choose an environment when a build is run. This prevents the ability for a single build to be used in multiple environments.
Environment should be optional.
For simplicity, email notifications will be removed from the the Hal Agent until the notification system is redesigned to support per-user, per-project, and per-environment preferences.
This is a new specification for standardizing where files are located in php packages.
Elastic Container Service is an amazon service that provides docker management and a method for deploying container groups.
We should refactor the permission system to allow better flexibility of both ACL and RBAC.
Should be able to create roles, and assign multiple user to the same role. A role is essentially a combination of one or more ACLs.
Should still be able to give users individual access.
There should be tiers of access, and scope that access is granted to. Scope is considered a limiter.
| Role | Tier | Scope |
|---|---|---|
| Super Admin | Super | Global (Can manage all global configuration) |
| Developer | Deploy |
(Can deploy single app to non-prod environments) |
| Release Engineer | Deploy | Global (Can deploy all apps and environments) |
| SRE/Ops | Deploy | Environment(prod) (Can deploy all apps to only prod) |
| Support Admin | Admin | Global (Can manage some global configuration) |
| Tech Lead | Owner | Organization(efg) (Can create and manage app configuration for all apps in org) |
Are tiers inclusive of lower levels? What operations do each grant? Does this differ by scope?
e.g. admin access to environments means different things from admin access to an app.
The operations granted by each scope x tier should be documented.
Is super tier necessary? Is member necessary?
There are other things to consider here since we are adding the ability to grant permission to an entire organization. Organizations are currently lightweight and have no meaning. Anyone can add an app to an org, and apps can move freely between them. We should model after github's behavior wrt permission when transferring a repo between orgs.
Also keep in mind self service and the ability of application creators/owners to grant others permission.
Can this system be expanded? What if we want to add tiers, or scopes? Are database migrations necessary for this?
In tandem with #44 (Add health status endpoint), we should be able to run a test against the hal instance that traverses the entire API to ensure every endpoint returns an OK response. This is pretty easy to do since the API is hypermedia-based and a single test could crawl the entire API from the index.
This test could be run on a schedule to ensure hal is healthy and still alive.
Hal should have a status endpoint for the UI and API.
Github API package updated to 2.0 which supports newer versions of guzzle - https://github.com/KnpLabs/php-github-api
Several options exist for adding support to building windows apps. Hal already supports deploying windows apps to AWS.
The frontend css is a bit of a mess and hasn't given proper attention in a while. This is a technical debt refactor and clean-up.
This includes updating fixing property names (removing old ones), changing resource names, and otherwise fixing the domain model and entities.
Current entities:
- Application
- AuditLog
- Build
- Credential
- AWSCredential
- PrivateKeyCredential
- Deployment
- DeploymentPool
- DeploymentView
- EncryptedProperty
- Environment
- EventLog
- Group
- Process
- Push
- Server
- Token
- User
- UserPermission
- UserSettings
- UserType
yml
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.