tf serving using mkl dnn has longer lantecny

[liangzimei]

tf serving using mkl dnn has longer lantecny

[liangzimei]

Then i set TensorFlow Serving Configuration Settings following GeneralBestPractices. The difference becomes smaller, but sering with mkl-dnn still performs worse.

[karthikvadla]

@liangzimei Can you please let me know how you are building TFserving image, which version you are using? Tensorflow Serving == 1.13 or master? With master we some performance issues.

[karthikvadla]

@liangzimei can you share me exact script you used to track the latency, i'm trying to reproduce same at my end.

[karthikvadla]

Hi @liangzimei , I tested out below two scenarios with latest master branch of tensorflow serving repo.

NOTE: Currently master branch is broken with builtin modules missing error. Ebi, created a patch to fix it. So it did use patch to build docker images.

Clone Tensorflow Serving

git clone [email protected]:tensorflow/serving.git
cd serving
git pull origin pull/1367/head

Build MKL-DNN Docker images and Test

I followed instructions from here to test ResNet50-V1 model.

1. Build image

cd tensorflow_serving/tools/docker

docker build \
    --build-arg HTTP_PROXY=${HTTP_PROXY} \
    --build-arg HTTPS_PROXY=${HTTPS_PROXY} \
    --build-arg http_proxy=${http_proxy} \
    --build-arg https_proxy=${https_proxy} \
    -f Dockerfile.devel-mkl -t tensorflow/serving:latest-devel-mkl .

docker build \
--build-arg HTTP_PROXY=${HTTP_PROXY} \
--build-arg HTTPS_PROXY=${HTTPS_PROXY} \
--build-arg http_proxy=${http_proxy} \
--build-arg https_proxy=${https_proxy} \
-f Dockerfile.mkl -t tensorflow/serving:mkl .

2. Get NCHW pre-trained model

NOTE: NCHW data format is optimal for Intel-optimized TensorFlow Serving.

mkdir /tmp/resnet

curl -s http://download.tensorflow.org/models/official/20181001_resnet/savedmodels/resnet_v1_fp32_savedmodel_NCHW_jpg.tar.gz \
| tar --strip-components=2 -C /tmp/resnet -xvz

3. Run Tensorflow model server for the downloaded model using built image
   Tested on machine with 2-sockets and 28 cores each. Below is the optimal configuration for this model. Setting 1/4th of physical cores gave optimal performance.

docker run \
  -p 8501:8501 \
  --name=tfserving_resnet_restapi \
  -v "/tmp/resnet:/models/resnet" \
  -e OMP_NUM_THREADS=14 \
  -e TENSORFLOW_INTER_OP_PARALLELISM=2 \
  -e TENSORFLOW_INTRA_OP_PARALLELISM=14 \
  -e MODEL_NAME=resnet \
  tensorflow/serving:mkl &

4. Download client and Send predict requests

Download client

 curl -o /tmp/resnet/resnet_client.py https://raw.githubusercontent.com/tensorflow/serving/master/tensorflow_serving/example/resnet_client.py

Send Request

 for i in {1..10};do python /tmp/resnet/resnet_client.py; done

Result:

Prediction class: 286, avg latency: 43.8565 ms
Prediction class: 286, avg latency: 43.6785 ms
Prediction class: 286, avg latency: 45.0794 ms
Prediction class: 286, avg latency: 44.5519 ms
Prediction class: 286, avg latency: 44.6467 ms
Prediction class: 286, avg latency: 44.8927 ms
Prediction class: 286, avg latency: 45.3638 ms
Prediction class: 286, avg latency: 44.973 ms
Prediction class: 286, avg latency: 45.6599 ms
Prediction class: 286, avg latency: 45.21 ms

Build Eigen Docker images and Test

I followed same instructions as above. Only change is we used NHWC data format with Eigen

1. Build images

docker build \
    --build-arg HTTP_PROXY=${HTTP_PROXY} \
    --build-arg HTTPS_PROXY=${HTTPS_PROXY} \
    --build-arg http_proxy=${http_proxy} \
    --build-arg https_proxy=${https_proxy} \
    -f Dockerfile.devel -t tensorflow/serving:latest-devel .

docker build \
--build-arg HTTP_PROXY=${HTTP_PROXY} \
--build-arg HTTPS_PROXY=${HTTPS_PROXY} \
--build-arg http_proxy=${http_proxy} \
--build-arg https_proxy=${https_proxy} \
-f Dockerfile -t tensorflow/serving:eigen .

2. Get pretrained model

mkdir /tmp/resnet2 

curl -s http://download.tensorflow.org/models/official/20181001_resnet/savedmodels/resnet_v1_fp32_savedmodel_NHWC_jpg.tar.gz \
| tar --strip-components=2 -C /tmp/resnet2 -xvz

3. Run docker container, before start container make sure to stop previous container.

docker rm -f tfserving_resnet_restapi

docker run \
  -p 8501:8501 \
  --name=tfserving_resnet_restapi \
  -v "/tmp/resnet2:/models/resnet" \
  -e MODEL_NAME=resnet \
  tensorflow/serving:eigen &

4. Send Request

for i in {1..10};do python /tmp/karthik/git-repo/tfserving/resnet/resnet_client.py; done

Result

Prediction class: 286, avg latency: 79.6406 ms
Prediction class: 286, avg latency: 77.7406 ms
Prediction class: 286, avg latency: 72.6479 ms
Prediction class: 286, avg latency: 71.69 ms
Prediction class: 286, avg latency: 74.3118 ms
Prediction class: 286, avg latency: 74.3501 ms
Prediction class: 286, avg latency: 72.5633 ms
Prediction class: 286, avg latency: 81.9913 ms
Prediction class: 286, avg latency: 80.9617 ms
Prediction class: 286, avg latency: 74.1507 ms

As you can see, MKL has less latency compared to Eigen.

[karthikvadla]

Closing this issue, we don't see any problems. Please feel free to open, if you still see issue. Please provide complete steps to re-produce.