deepkit/deepkit-python-sdk

Deepkit

Deepkit is an open-source platform and cross-platform desktop application to execute, track, and debug modern machine learning experiments. With integrated server management tool to execute experiments on any Linux server with a simple click or cli command.

Download - Example Python

Features

• Experiment execution using uniform experiment definition
• Execute experiments locally or in the cloud, seamlessly
• Pipeline support to increase execution time and reduce hardware costs
• Layer debugger for Keras2 and Pytorch
• Automatic source code versioning
• Experiment artifacts (aka output: weights, snapshots, etc)
• Experiment execution via Docker container with auto-build
• Experiment comparison: Side by side, file diff, metrics diff
• Server job scheduling with resource limitation and assignment (and NVIDIA Docker GPU support)
• Real-time experiment tracking: Log metrics, text, images, numpy arrays, and more
• Python SDK
• Project management: Issue tracker, Kanban board, and notes
• Real-time desktop (and web) app with analytical tools to monitor and organize experiments
• Optional centralized server for teams

Development

Deepkit is written entirely in Typescript and consists of following main packages:

• packages/cli: command line interface tools
• packages/deepkit: The GUI, desktop and web app interface
• packages/server: The server component that stores all the data using the filesystem and mongodb
• packages/website: The website as SSR Angular 9 SPA app
• packages/core: Common models, functions, services
• packages/core-node: Common models, functions, services for node environment
• packages/electron: The electron bootstrapper for the GUI

Following open-source packages have been created for Deepkit that wouldn't exist otherwise and were absolutely necessary:

• Marshal.ts - The fastest universal Typescript data serializer and validator
• Glut.ts - A reactive real-time server framework with distributed entity/file abstraction and syncing based on ReactJS
• angular-desktop-ui - A GUI library for native-like GUI widget based on Angular 9+

Development app

cd deepkit
npm run bootstrap

cd deepkit
npm-local-development

cd deepkit
make angular-watch

cd deepkit
server-watch

# Open in browser the angular dist
# e.g. file:///Users/marc/bude/deepkit/packages/deepkit/dist/deepkit/index.html
# or electron
cd deepkit
make electron-build
make electron

Development website

Watcher

cd packages/website
ng run website:server:production --watch

Server

cd packages/website
npm run serve:ssr

deepkit-python-sdk's Issues

No hardware resource monitoring in some cases

When I launch an experiment from the Deepkit "Start" dialog using a yml config, I don't see any hardware resource monitoring metrics. I do see metrics when either launching a remote docker experiment from the Deepkit "Start" dialog or when launching locally by "Script run".

Setup:

• Windows 10
• Deepkit Release 2020.1.5
• Deepkit SDK 1.0.1

Update PyTorch Example

The PyTorch example could use some improvement:

• It doesn't reflect the latest API anymore: the experiment.log_metric1 signature has swapped xandy`.
• When using multiprocessing for data loading (that's common practice with PyTorch's torch.utils.data.DataLoader), its problematic to execute expressions outside an if __name__ == '__main__' clause (at least on Windows): it starts multiple experiments...
• I usually want to have my training script also be possible to launch from console. What do you think of the proposed snipped for merging both sets of arguments/options?

from collections import ChainMap
​
def main():
    # Training settings
    parser = argparse.ArgumentParser(description='PyTorch MNIST Example',
                                     formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    parser.add_argument('--batch-size', type=int, default=64, metavar='N',
                        help='input batch size for training')
    # More arguments...
    args = parser.parse_args()
    experiment = deepkit.experiment()
    # Merge CLI arguments (low priority) with experiment hyperparameters (high priority)
    args = argparse.Namespace(**ChainMap(experiment.full_config(), vars(args)))
    # Update merged arguments back to experiment server
    for key, val in vars(args).items():
        experiment.set_config(key, val)
    # Capsule the experiment handle into `args` as well, so distribution to train() and test() methods is easy
    args.experiment = experiment

watch_torch_model: only first training instance visible

watch_torch_model() does not work entirely. For all images, I somehow only see the very first training instance the model encountered, though the distributions and also some stuff in the weights still seem to get updated.

Setup:

• Windows 10
• Deepkit Release 2020.1.5
• Deepkit SDK 1.0.1
• Running experiment on anything, from local script run, to local UI start to deployed docker run, eg. on pytorch/pytorch:1.4-cuda10.1-cudnn7-runtime

No STDOUT logs on Windows

When using the account localhost, STDOUT print logs do not appear in any case (launching from console, starting from Deepkit UI or starting in remote docker).

Setup:

• Windows 10
• Deepkit Release 2020.1.5
• Deepkit SDK 1.0.1
• Python 3.7.x (local and remote)

They start showing up, however, when the Deepkit App is connected to a remote-hosted Team Server (eg. a non-localhost account)