umass-rescue/mldatasettemplate

Template for hosting a dataset and remotely training models on the data

mldatasettemplate's Issues

Create logs of worker console output

When a worker trains a model, it creates a lot of logs with important information and potentially important error messages. Log all outputs to a unique file tied to the training job instead of stdout on the worker.

Allow for dynamic loading of loss functions

Right now there is no way for a user to specify a loss function. Unlike optimizers which can be specified with a string loss functions must be loaded directly from a python object. Determine a safe and usable way for researchers to specify their own loss functions on upload

Constant Image Sizing on Load

Currently, datasets can contain images of any size. On load the images should be modified to be the same size as to work with all models.

Enable Worker Scaling

Similar to this Server Issue which also handles Docker worker scaling.

Right now, there is a single Docker worker which handles all training tasks. This is not optimal, as the single worker is hard-coded into the docker-compose.yml file and will not handle large request loads. Figure out the best way to dynamically scale the number of workers available.

Support Pytorch Lightning

Right now all training is done via Tensorflow2 and Keras. Add support for Pytorch Lightning.

Enable GPU For Training

Currently, the Docker container used for the microservice only has access to the CPU and memory. Enabling the container to interact with the system GPU can lead to large speed ups in training tasks.

Handle Automatic Train/Test Split

Right now the server only supports a train/validation split with the entire dataset, and no test set is generated when the dataset is loaded.

Create a new environment variable in .env for train/test split, and then use this new test set to provide additional statistics on model training.