Prerequisites

Bazel

Using Bazel custom APT repository (recommended)

  • Install JDK 8
$ sudo apt-get install openjdk-8-jdk
  • Add Bazel distribution URI as a package source (one time setup)
$ echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
$ curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add

If you want to install the testing version of Bazel, replace stable with testing.

  • Install and update Bazel
$ sudo apt-get update && sudo

Once installed, you can upgrade to a newer version of Bazel with:

$ sudo

gRPC Python

$ sudo

Packages dependencies

$ sudo apt-get update && sudo apt-get install -y \
        build-essential \
        curl \
        libcurl3-dev \
        git \
        libfreetype6-dev \
        libpng12-dev \
        libzmq3-dev \
        pkg-config \
        python-dev \
        python-numpy \
        python-pip \
        software-properties-common \
        swig \
        zip \

TensorFlow Serving Python API PIP package

$ pip install tensorflow-serving-api

Installing from source

Clone the TensorFlow Serving repository

$ git clone --recurse-submodules https://github.com/tensorflow/serving

–recurse-submodules is required to fetch TensorFlow, gRPC, and other libraries that TensorFlow Serving depends on.
Note that these instructions will install the latest master branch of TensorFlow Serving. If you want to install a specific branch (such as a release branch), pass ​​​-b <branchname>​​ to the git clone command.

Install prerequisites

Follow the Prerequisites section above to install all dependencies. To configure TensorFlow, run

$ cd tensorflow
$ ./configure
$ cd ..

Build

$ bazel build -c opt tensorflow_serving/...

Binaries are placed in the bazel-bin directory, and can be run using a command like:

$ bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server

To test your installation, execute:

$ sudo pip install autograd # handle ImportError: No module named autograd
$ bazel test -c opt tensorflow_serving/...

Serving a TensorFlow Model

Train And Export TensorFlow Model

  • step 0
$ rm -rf /tmp/mnist_model
  • step 1
$ bazel build -c opt //tensorflow_serving/example:mnist_saved_model
$ bazel-bin/tensorflow_serving/example/mnist_saved_model /tmp/mnist_model
Training model...

...

OR

$ python tensorflow_serving/example/mnist_saved_model.py /tmp/mnist_model

list the model files

$ ls /tmp/mnist_model
1
$ ls /tmp/mnist_model/1

Each version sub-directory contains the following files:

  • saved_model.pbis the serialized tensorflow::SavedModel. It includes one or more
  • graphdefinitions of the model, as well as metadata of the model such as signatures.
    variables are files that hold the serialized variables of the graphs.

Load Exported Model With Standard TensorFlow ModelServer

$ bazel build -c opt //tensorflow_serving/model_servers:tensorflow_model_server # do not use
$ bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --port=9000 --model_name=mnist --model_base_path=/tmp/mnist_model/

Test The Server

$ bazel build -c opt //tensorflow_serving/example:mnist_client
$ bazel-bin/tensorflow_serving/example/mnist_client --num_tests=1000 --server=localhost:9000
...
Inference error rate: 10.4%

OR

$ python tensorflow_serving/example/mnist_client.py --num_tests=1000 --server=localhost:9000

Reference