sxnjey/bailo

Managing the lifecycle of machine learning to support scalability, impact, collaboration, compliance and sharing.

Bailo is still in development and we have not yet completed all of the features we want it to have. See the roadmap for what we plan to build.

Making it easy to compliantly manage the machine learning lifecycle
Explore the docs »

Report a Bug · Request a Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Roadmap
4. Usage
5. Contributing
6. Breaking Changes
7. License
8. Acknowledgments

About The Project

Bailo helps you manage the lifecycle of machine learning to support scalability, impact, collaboration, compliance and sharing.

Built With

• Next.js
• Node.js
• MongoDB
• Seldon

Getting Started

Requirements:

• Node v16
• Docker / Docker Compose

Installation:

To run in development mode (modified files on your host machine will be reloaded into the running application):

git clone https://github.com/gchq/Bailo.git && cd Bailo
npm install
npm run certs
sed "s/user_id=REPLACE_WITH_UID/user_id=$UID/" docker-compose-dev.yml > docker-compose-dev-personal.yml
docker-compose -f docker-compose-dev-personal.yml up --force-recreate --build -d

The creation of docker-compose-dev-personal.yml gives the application user the same uid as you on your host machine. This allows the application to run the app from the files on your host machine, and populate node_modules (via npm install).

On first run, it may take a while (perhaps 30 seconds) to start up. It needs to build several hundred TypeScript modules. These are cached however, so future starts only require a few seconds. There's also npm run dev2 for an alternative type checker that is more rigorous. You should access the site via localhost:8080 which provides authentication as a test user.

Setup:

Some example schemas are installed by default. More schemas can be added by altering and running the addDeploymentSchema.ts and addUploadSchema.ts files.

ts-node --project tsconfig.server.json server/scripts/addDeploymentSchema.ts
ts-node --project tsconfig.server.json server/scripts/addUploadSchema.ts

NOTE: Scripts are also written in Typescript, install ts-node with npm install -g ts-node.

Service Ports:

Service	Host	Notes
NodeJS App	3000	Internal only port
Nginx	8080	Access the UI via this
Mongo	27017	No credentials
Registry	5000	HTTPS only, no UI
Minio UI	9001	minioadmin:minioadmin
Minio	9000	minioadmin:minioadmin

** Note: these credentials are intentionally basic/default, but in your own instances we recommend changing them to something more secure.

Always connect to the project via 'Nginx', otherwise you will receive authentication failed errors. We expect the administrator to provide their own forms of authentication.

You can test out your new deployment using the example models which can be found in __tests__ minimal_binary.zip and minimal_code.zip. There are also example forms in the scripts folder minimal_upload_schema_examples.json and minimal_deployment_schema_examples.json.

Logical Project Flow (Overview)

1. A user accesses a URL. We use NextJS routing to point it to a file in pages. [xxx].tsx files accept any route, xxx.tsx files allow only that specific route.
2. Data is loaded using SWR. Data loaders are stored in ./data. Each one exposes variables to specify if it is loading, errored, data, etc.
3. Requests to the backend get routed through express within server/index.ts. Each route is an array with all items being middleware except the last, which is the handler ([...middleware, handler]).
4. Routes interact with the database via mongoose, which stores models in ./server/models.

Some processing is done away from the main thread, when it is expected to take longer than a few milliseconds. These are posted to a mongodb queue and processed by handlers in the server/processors folder. Mongodb queues are handled invisibly by p-mongo-queue (server/utils/queues.ts).

Known Issues

Issue: Sometimes Docker struggles when you add a new dependency.

Fix: Run docker-compose down --rmi all followed by docker-compose up --build.

Issue: Sometimes SWR fails to install its own binary and the project will refuse to start up (development only)

Fix: Run npm uninstall next && npm install next. Some users report still having issues. If so, run: rm -rf node_modules && rm -rf package-lock.json && npm cache clean -f && npm i.

Issue: Unable to authenticate to the Docker registry / compile binaries.

Fix: Make sure that your authentication proxy is setup to allow the 'Authorisation' header. Make sure that your application is able to access the Docker registry internally as it will not provide user authentication.

Roadmap

List of near term goals:

• Python client
• Improve test coverage of core deployment
• K8s Helm charts
• AWS deployment pattern
• Azure deployment pattern
• Deployment container watermarking

Usage

See our user documentation

Contributing

See our contribution guide

Breaking Changes

28-04-2022: Changed 'schema' model to be stored as a string instead of an object. Delete & recreate your schemas.

License

Bailo is released under the Apache 2.0 Licence and is covered by Crown Copyright. See public/LICENSE.txt for more information.

Acknowledgments

• Img Shields
• Othneils's README Template
• Mattermost's Code Contribution Guidelines