Updated for CUDA 11 and Tensorflow 2!!!
This repository allows you to get started with training a State-of-the-art Deep Learning model with little to no configuration needed! You provide your labeled dataset and you can start the training right away and monitor it with TensorBoard. You can even test your model with our built-in Inference REST API. Training with TensorFlow has never been so easy.
You can also use our BMW-Labeltool-lite to label your dataset. The images and labels can be used directly for training.
- This repository is based on Tensorflow Object Detection API
- The tensorflow version used is in this repo is 2.5.0
- All supported networks in this project are taken from the tensorflow model zoo
- All training are made using pre-trained network weights.
- The pre-trained weights that you can use out of the box are based on the COCO dataset.
- The app was tested with Google Chrome and it is recommended to use Chrome when training.
- This repository support training on both CPU and on multiple GPU (up to 2 GPUs)
- Ubuntu 18.04
- NVIDIA Drivers (418.x or higher)
- Docker CE latest stable release
- NVIDIA Docker 2
- Docker-Compose
This step is recommended to be able to run the solution correctly.
The setup script will check and adjust all the requirements needed based on the user input.
-
Run the following command
chmod +x setup_solution_parameters.sh && source setup_solution_parameters.sh-
The script will check if docker and docker-compose are installed and if not, it will install them.
-
You will be prompted to choose build architecture GPU/CPU for the training solution.
- If you chose GPU the script will check if nvidia-docker is installed and if not, it will install it.
-
The script will prompt all the network interfaces that are available so you can select which interface you want to extract the ip address from it
-
You will be prompted to choose if you want to set up a proxy
-
You will be prompted to select all the pre-trained weights network that you want to be downloaded during the docker image build (use the up/down arrow to go up and down, space to select/unselect, enter to submit your selection and esc to quit)
-
To check if you have docker-ce installed:
docker --version
To check if you have docker-compose installed:
docker-compose --version
To check if you have nvidia-docker installed:
dpkg -l | grep nvidia-docker
To check your nvidia drivers version, open your terminal and type the command nvidia-smi
-
If you don't have neither docker nor docker-compose use the following command
chmod +x install_full.sh && source install_full.sh -
If you have docker ce installed and wish only to install docker-compose and perform necessary operations, use the following command
chmod +x install_compose.sh && source install_compose.sh -
Install NVIDIA Drivers (418.x or higher) and NVIDIA Docker for GPU training by following the official docs
-
Make sure that the
.gitkeepfiles indatasets,checkpoints,tensorboardsandinference_api/modelsfolder are deleted. (.gitkeep files are placeholder files used for git) -
Make sure that the
base_dirfield indocker_sdk_api/assets/paths.jsonis correct (it must match the path of the root of the repo on your machine).
-
Make sure that the
image_namefield indocker_sdk_api/assets/paths.jsonis correct (it must match your chosen architecture for the trainingtf2_training_api_cpuortf2_training_api_gpu).
-
Go to
gui/src/environments/environment.tsandgui/src/environments/environment.prod.tsand change the following:-
field
url: must match the IP address of your machine -
the IP field of the
inferenceAPIUrl: must match the IP address of your machine (Use theifconfigcommand to check your IP address . Please use your private IP which starts by either 10. or 172.16. or 192.168.)
environment.ts
environment.prod.ts
-
If you are behind a proxy:
-
Enter you proxy settings in the
<base-dir>/proxy.jsonfile
-
Enter the following command:
python3 set_proxy_args.py
The following is an example of how a dataset should be structured. Please put all your datasets in the datasets folder.
├──datasets/
├──sample_dataset/
├── images
│ ├── img_1.jpg
│ └── img_2.jpg
├── labels
│ ├── json
│ │ ├── img_1.json
│ │ └── img_2.json
│ └── pascal
│ ├── img_1.xml
│ └── img_2.xml
└── objectclasses.jsonPS: you don't need to have both json and pascal folders. Either one is enough
- If you want to label your images, you can use our BMW-LabelTool-Lite which is a free, open-source image annotation tool. This tool supports our JSON label format
You must include in your dataset an objectclasses.json file with a similar structure to the example below:
Midweight: Downloading specific supported online pre-trained weights during the docker image build.
To do that, open the json file training_api/assets/networks.json and change the values of the networks you wish to download to true.
Heavyweight (default): Downloading all the supported online pre-trained weights during the docker image build.
To do that, open the json file training_api/assets/networks.json and change the value of "select_all" to true.
PS: if you don’t download the weights during the build, you won’t be able to use the network during the training unless you rebuild the solution with the proper network chosen.
All the training are made using the pre-trained network weight based on coco dataset.
If you wish want to deploy the training workflow in GPU mode, please write the following command from the repository's root directory
docker-compose -f build_gpu.yml buildIf you wish want to deploy the training workflow in CPU mode, please write the following command from the repository's root directory
docker-compose -f build_cpu.yml build If you wish want to deploy the training workflow in GPU mode, please write the following command
docker-compose -f run_gpu.yml upIf you wish to deploy the training workflow in CPU mode, please write the following command
docker-compose -f run_cpu.yml upAfter a successful run you should see something like the following:
-
If the app is deployed on your machine: open your web browser and type the following:
localhost:4200or127.0.0.1:4200 -
If the app is deployed on a different machine: open your web browser and type the following:
<machine_ip>:4200
Prepare your dataset for training
Specify the general parameters for you docker container
Specify the hyperparameters for the training job
Specify the advanced hyperparameters for the training job
Check your training logs to get better insights on the progress of the training
Monitor the training using Tensorboard
Check the status to know when the job is completed successfully
Download your mode and easily test it with the built-in inference API using Swagger
Delete the container's job to stop an ongoing job or to remove the container of a finished job. (Finished jobs are always available to download)
Visualize graphs and metrics of Deleted Jobs with Tensorboard
Check our tips document to have (1) (a better insight on training models based on our expertise) and (2) (a benchmark of the inference speed).
Our tensorboard document helps you find your way more easily while navigating tensorboard
-
In advanced configuration mode, be careful while making the changes because it can cause errors while training. If that happens, stop the job and try again.
-
the paths in
fine_tune_checkpointshould be entered manually when you choose train from checkpoint -
Scroll down to find fine_tune_checkpoint and replace with your network-name and checkpoint name:
-
fine_tune_checkpoint :
fine_tune_checkpoint: "/checkpoints/<network-name>/<name-of-the-checkpoint>/ckpt-0"
-
-
-
In general settings, choose carefully the container name because choosing a name used by another container will cause errors.
-
When you try to monitor the job using tensorboard the page may not open, wait for some seconds and refresh the page.
-
When you leave tensorboard open for a long time, it might freeze. When encountered with such issue simply closing tensorboard tab in the browser and reopening it will solve the problem.
To change the docker-sdk default port 2222 to any other port of your choice:
-
change the uvicorn port inside
docker_sdk_api/docker/Dockerfileto the port of your choice
-
rebuild the docker-sdk image using the following commend in the root of the repo:
docker-compose -f build.yml build docker_sdk
-
change the
baseEndPoint : <port-of-your-choiceinsidegui/src/environments/environment.tsandgui/src/environments/environment.prod.ts
-
rebuild the GUI image using the following comment in the root of the repo:
docker-compose -f build.yml build user_interface
-
after this you can run the solution as following:
docker-compose -f run.yml up
You might face some errors in some cases during the training. Most common ones are:
The running container has no RepoTag please kill to proceed: Container ID: <id-container>This issue is caused by some container not having a name, in that case you should rename that container or kill (make sure it is safe to remove this container) it viadocker kill <id-container>.Job Not Started: 404 Client Error Not Found("pull access denied for <image-name>, repository does not exists or may require 'docker login' ...)this issue is cause when you are trying to run a training docker image that you don't have. The main reason of this is not properly building the training_api or not setting up project requirements please refer to Setting Up Project Requirements section in the documentation.Dataset Not Validthis error means that you dataset structure is not valid or the images/labels formate are not supported.- Training job not started after general settings step: One of the main reason is that the paths are not adjusted in
docker_sdk_api/assets/paths.jsonfieldbase_dir. You can solve this issue by running./setup_solution_parameters.shand choosing the training version you want to useGPU/CPU.
If you use this repository in your research, consider citing it using the following Bibtex entry:
@misc{bmwtrainingtool,
author = {BMW TechOffice MUNICH},
title = {TensorFlow Training GUI},
howpublished = {\url{https://github.com/BMW-InnovationLab/BMW-TensorFlow-Training-GUI}},
year = {2022},
}
-
Hadi Koubeissy, inmind.ai, Beirut, Lebanon
-
Ismail Shehab, inmind.ai, Beirut, Lebanon
-
Joe Sleiman, inmind.ai, Beirut, Lebanon
-
Jimmy Tekli, BMW Innovation Lab, Munich, Germany
-
Chafic Abou Akar, BMW TechOffice, Munich, Germany
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent