Reusing Frozen Inference Models in Tensorflow

Recently I've been doing a lot of Tensorflow research at work and have found that trying to swap out different models for the tutorials as to make the software your own can be somewhat difficult without doing hours and hours of reading, and today I hope to correct that.

What Did I Do?

This article will hopefully culminate a combination of all of the following tutorials/projects:

And these not projects, but still important sources

Installation and Setup
Training the mobilenet model
Creating a frozen inference graph
Reusing that graph in a different application

Installation and Setup

If you haven't installed tensorflow yet, take a look over here for a walkthrough of the installation process to ensure that you get all of the dependencies installed

Required Repositories

There are a couple repositories required for this project, if you followed the install guide supplied, then you should have git installed, otherwise you can use the Github Desktop to aid in the installation, but if you want a terminal version, then you'll want to install Git. Typically I put all of my github repositories into their own folder, like C:\Github_Repos\. You'll want to put both the tensorflow and tensorflow/models repositories into the same folder, at least I did.

First things first, here's the list of repositories to download:

I've written a shell script to automate the process of setting up the repositories, since Tensorflow's Models repository (specifically the object_detection repo) actually requires a bit of installation, you can follow the code below, or you can visit here to find the entire thing available for copy and paste, simply replace all instances of [GITHUB REPOSITORY LOCATION] with your github repo directory. Or you can also follow Tensorflow's official installation page here.

Git

If you don't have git, then you need to install git.

sudo apt install git

Pip packages

Next there is a whole slew of pip packages required, you can either use pip or pip3, it depends on the python version you wish to use for the rest of the project

sudo apt-get install protobuf-compiler pip3 install tensorflow pip3 install --user Cython pip3 install --user contextlib2 pip3 install --user pillow pip3 install --user lxml pip3 install --user jupyter pip3 install --user matplotlib pip3 install --user opencv-python

Github Repositories

Next, you'll want to move to your github repository location to download the repositories, replace [GITHUB REPOSITORY LOCATION] with your desired github repo directory

cd "[GITHUB REPOSITORY LOCATION]" git clone https://github.com/cocodataset/cocoapi.git mkdir tensorflow cd tensorflow git clone https://github.com/tensorflow/models.git

Coco

Now we need to build the Coco Python API for tensorflow to use

cd ../cocoapi/PythonAPI make cp -r pycocotools ../../tensorflow/models/research

Compiling the Protobufs

Now we need to compile tensorflow's object_detection api's protobufs

cd ../../tensorflow/models/research protoc object_detection/protos/*.proto --python_out=.

Compiling the Object Detection library for use

I know it seems like we already did this, but we need to execute a couple more scripts in order for things to work, you can either use python, or python3

python3 setup.py build python3 setup.py install

Environment variables!

No walkthrough is complete without adding some variables to the environment, if you don't know how to do this on windows, review here. Just take the variables that we make here and apply them to windows, for linux, do the following:

py_path="$(pwd)" export PYTHONPATH=$PYTHONPATH:${py_path}:${py_path}"/slim" sudo sh -c 'echo "export PYTHONPATH=$PYTHONPATH:${py_path}:${py_path}/slim" >> ~/.bashrc'

Repository Installed!

That should be it, if you did everything correctly, then when you test the installation with:

python3 object_detection/builders/model_builder_test.py

The script should return no output

Setting up the Pet Detection Project

I wrote a couple of scripts for this, which you can find here, or just run the code below with [GITHUB DIRECTORY HERE] changed to your desired github repo directory.

First things first, you need to switch to your desired installation directory ie. C:\users\you\Documents\workspace\object_detection_tutorial\

cd "C:\users\you\Documents\workspace\object_detection_tutorial\"

Creating the project directories

Now, following this as a reference, setup a directory tree that mimics the one they used

mkdir "data" mkdir "models" mkdir "models/model" mkdir "models/model/train" mkdir "models/model/eval"

Downloading the datasets

Now we need to get the datasets used in the project

If you want to use custom datasets: then you need to modify the next couple of steps, you still download the pictures as stated below, but, when it comes to the label map and the TFRecords, you'll need to create your own annotations, which is described in detail here. running the create_pet_tf_record.py will not work either, you need to follow the steps here where I've written a tutorial specifically on how to create TFRecords for a custom dataset, if you do that, then you can skip to "The First Model".

If you followed the tutorial at 3sidedcube.com described above, then you'll note a section telling you how to convert your xml files into csv, for the next section on creating TFRecords, you don't need to do that, just leave the xml files in the annotations folder, and the create_pet_tf_record.py will take care of everything else.

wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/annotations.tar.gz tar -xf images.tar.gz tar -xf annotations.tar.gz

Label maps and TFRecords

If you're needing to create your own custom dataset, please refer to the previous section on collecting the data

Swipe yourself a copy of tensorflow's object_detection api's pet_label_map.pbtxt, remember to replace [GITHUB DIRECTORY HERE] with the directory that you downloaded the github repositories into. If you are using your own dataset, then you'll need a custom label map, for information on creating label maps, refer to here and/or object_detection/data/pet_label_map.pbtxt for examples.

cp "[GITHUB DIRECTORY HERE]/tensorflow/models/research/object_detection/data/pet_label_map.pbtxt" "data"

Now we need to create TFRecords using the images given and the label map. If you want to store the images elsewhere or you are using a different dataset, then you can specify that using the --data_dir directive, you can also specify where the output records are stored using the --output_dir directive. If you are using a custom label map, then you need to change the --label_map_pathdirective to point to your label map .pbtxt file.

p="[GITHUB DIRECTORY HERE]/tensorflow/models/research/object_detection/dataset_tools/create_pet_tf_record.py" sudo python3 $p --label_map_path="./data/pet_label_map.pbtxt" --data_dir='.' --output_dir='./data'

The First Model

Okay, so now we're ready to download the model, in actuality, the training script used for this project will work with any of the models from Tensorflow's Object Detection Model Zoo, but the trick is, that you need a corresponding .config file or the model will fail during training, or train the model and accomplish nothing and waste enormous amounts of time (trust me, I'm speaking from experience here). So it's really important to get the model and corresponding .config files together.

Model

The process for downloading a model is pretty simple, simple download, unzip it, and move the .ckpt files to your model directory

model_name="ssdlite_mobilenet_v2_coco_2018_05_09" wget "http://storage.googleapis.com/download.tensorflow.org/models/object_detection/${model_name}.tar.gz" tar -xvf "${model_name}.tar.gz" -C ./models cp "./models/${model_name}/*.ckpt" ./models/model

Config

Now we need to edit the corresponding .config config file for this model, which we can find here: /object_detection/samples/configs/, these config files come with PATH_TO_BE_CONFIGURED locations which must be swapped out with your project directory, this can be automated with the sed command on linux (windows, I'm sorry, you're out of luck, Ctrl+F your way to victory my friend). If you'd like to create your own config file, I'd suggest using the one below and the faster_rcnn_resnet101_pets.config file as a reference and consult here.

data_dir="$(pwd)""/data" p="[GITHUB DIRECTORY HERE]/tensorflow/models/research/object_detection/samples/configs/ssdlite_mobilnet_v2_coco.config" cp "$p" models/model/ sed -i "s|PATH_TO_BE_CONFIGURED|${data_dir}|g" models/model/ssdlite_mobilnet_v2_coco.config sed -i "s|${data_dir}/model.ckpt|$(pwd)/models/model/model.ckpt|g" models/model/ssdlite_mobilnet_v2_coco.config

If you did all of this right, then you should be ready to move onto the next step, learning how to train the model!

Training the Mobilenet Model

If you've done everything above, then it's time to train the model (you might want to double check that you've done everything above). Depending on the model you chose, this can take anywhere from about an hour, to a few days, so be prepared, and don't get too frustrated if it doesn't work the first time, it's a learning process, go back and try again, most likely, you've messed up the .config file and it's incompatible with the model you're training, or maybe you just made a typo, I'd start there.

To train the model only takes one command according to here, and for those of you to lazy to click on it, here it is again:

# From the tensorflow/models/research/ directory PIPELINE_CONFIG_PATH={path to pipeline config file} MODEL_DIR={path to model directory} NUM_TRAIN_STEPS=50000 SAMPLE_1_OF_N_EVAL_EXAMPLES=1 python object_detection/model_main.py \ --pipeline_config_path=${PIPELINE_CONFIG_PATH} \ --model_dir=${MODEL_DIR} \ --num_train_steps=${NUM_TRAIN_STEPS} \ --sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES \ --alsologtostderr

If you get an error talking about not being able to pickle dict_values, TypeError: can't pickle dict_values objects. Then you need to go and edit some source code according to this source. In the models/research/object_detection/model_lib.py file around line 381, you need to change the following:

# Eval metrics on a single example. eval_metric_ops = eval_util.get_eval_metric_ops_for_evaluators( eval_config, category_index.values(), eval_dict)

# Eval metrics on a single example. eval_metric_ops = eval_util.get_eval_metric_ops_for_evaluators( eval_config, list(category_index.values()), eval_dict)

If you are using windows, you can't use the '\' things, you'll need to make it all into one line.

While training the script will create a series of .ckpt files according to the current iteration that it's on, see below:

A ckpt file is generated for the 0th iteration
A ckpt file is generated every 10 minutes during training
A ckpt file is generated when the training finishes
Each "checkpoint" during training consists of 3 files, model.ckpt.data-00000-of-00001, model.ckpt.meta, and model.ckpt.index. For distinctions between the three, see here.

Checkpoints are created as follows:
The first checkpoint is created as model.ckpt-0.
The subsequent checkpoints (one is generated every 10 minutes) are created as model.ckpt-{1, 2, 3,...,n-1} where n is the number of training iterations desired. The actual number depends on the current iteration number, it does not actually include every iteration, but instead a snapshot of whatever iteration the training is on at the time.
The last checkpoint will be model.ckpt-n where n is the number of training iterations.

You can also launch tensorboard at this point in either a different terminal, or by calling it before the above command and using the '&' directive after (for linux)

tensorboard --log_dir="./models/model"

Or with the '&' directive (only for linux)

tensorboard --log_dir="./models/model" &

IF YOUR TRAINING SESSION HANGS: you can use Ctrl+c to break it, this is true for any program or script, just an fyi

Okay, so if you're training was successful, or at least you think it is, then you can move to the freezing stage

Creating a Frozen Inference Graph From a Trained Model

And now we have a beautiful model, that's ready to be frozen! And the beautiful thing is, despite what other people say, you can do this with one line too! There exists a file in object_detection called export_inference_graph.py and all you have to do to make your graph is to call it from the command line with the following arguments:

python "[GITHUB DIRECTORY HERE]\tensorflow\models\research\object_detection\export_inference_graph.py" \ --input_type image_tensor \ --pipeline_config_path "./models/model/[YOUR CONFIG FILE NAME HERE].config" \ --trained_checkpoint_prefix "./models/model/model.ckpt" \ --output_directory "./models/model/frozen_graph"

The important things to note here are, the input_type, which according to the documentation can be any of 3 different values

image_tensor: "Accepts a uint8 4-D tensor of shape [None, None, None, 3]"
encoded_image_string_tensor: "Accepts a 1-D string tensor of shape [None] containing encoded PNG or JPEG images. Image resolutions are expected to be the same if more than 1 image is provided.
tf_example: "Accepts a 1-D string tensor of shape [None] containing serialized TFExample protos. Image resolutions are expected to be the same if more than 1 image is provided.

Depending on the input type of the model you selected, you should change it accordingly, if you are unsure, use image_tensor.

The trained_checkpoint_prefix which is the prefix to the model you are trying to freeze, see here:

To use the 50000th checkpoint file generated by the training method, then you'd set the prefix to "./models/model/model.ckpt-50000"

The script will then output a series of frozen inference graph files wherever the --output_directory is specified to (in the above case, the files are output to a frozen_graph folder in the model directory.

And that's it! If nothing went wrong there, then your inference graph should be ready to go!

Reusing the Frozen Inference Graph in a Project

So you think you're ready to use your newly trained model in a snappy new program, huh? Well we'll see, we'll see...

Alright, so first thin you've gotta do is download this github repository:

git clone "https://github.com/yeephycho/tensorflow-face-detection.git" cd tensorflow-face-detection

If you want something to compare your model to you can go ahead and open a command line and run the inference_usbCam_face.py, don't forget to pass in the device id of your camera, if you don't know what to use, just use 0

python "./inference_usbCame_face.py" 0

It opens a window showing the output of your webcam, and detects faces if there is one present, you can see an example here, there's supposed to be a shortcut key to stop the program, but I couldn't get it to work when I used it, so I just use Ctrl+c

You'll notice that there's a folder called model head in there, and you'll see the current model being used. You wanna replace this one with the one that you just exported, if you downloaded the model from somewhere, then you wanna find a .pb file and rename it to match the filename of this one, then replace the current one with yours.

With your new model installed, go ahead and run the inference_usbCam_face.py script again and take a look at the comparison! Look, your model is running!

That's it, you're done! You've successfully trained a model to detect pets, frozen it, then used that frozen graph somewhere else in an unrelated project! And you don't have to stop there, you can use that frozen graph anywhere, you just have to load it into tensorflow.

If you made it all the way down here, and something isn't working, or you just want to show your support, drop me a comment, or an email, I'll help you out. Peace out.

Search This Blog

Aaron's Blog

Total Pageviews