TensorBoard Tutorial: TensorFlow Graph Visualization

What is TensorBoard?

TensorBoard is the interface used to visualize the graph and other tools to understand, debug, and optimize the model. It is a tool that provides measurements and visualizations for machine learning workflow. It helps to track metrics like loss and accuracy, model graph visualization, project embedding at lower-dimensional spaces, etc.

TensorFlow Graph Visualization using Tensorboard Example

The image below comes from the TensorBoard graph you will generate in this TensorBoard Tutorial. It is the main panel:

From the picture below, you can see the panel of TensorBoard graph visualization. The panel contains different tabs, which are linked to the level of information you add when you run the model.

Scalars: Show different useful information during the model training
Graphs: Show the model
Histogram: Display weights with a histogram
Distribution: Display the distribution of the weight
Projector: Show Principal component analysis and T-SNE algorithm. The technique used for dimensionality reduction

During this TensorBoard Tutorial, you will train a simple deep learning model. You will learn how it works in a future tutorial.

If you look at the graph, you can understand how the model work.

Enqueue the data to the model: Push an amount of data equal to the batch size to the model, i.e., Number of data feed after each iteration
Feed the data to the Tensors
Train the model
Display the number of batches during the training. Save the model on the disk.

The basic idea behind tensorboard is that neural network can be something known as a black box and we need a tool to inspect what’s inside this box. You can imagine tensorboard as a flashlight to start dive into the neural network.

It helps to understand the dependencies between operations, how the weights are computed, displays the loss function and much other useful information. When you bring all these pieces of information together, you have a great tool to debug and find how to improve the model.

To give you an idea of how useful the TensorBoard graph can be, look at the picture below:

TensorBoard graph

A neural network decides how to connect the different “neurons” and how many layers before the model can predict an outcome. Once you have defined the architecture, you not only need to train the model but also a metrics to compute the accuracy of the prediction. This metric is referred to as a loss function. The objective is to minimize the loss function. In different words, it means the model is making fewer errors. All machine learning algorithms will repeat many times the computations until the loss reaches a flatter line. To minimize this loss function, you need to define a learning rate. It is the speed you want the model to learn. If you set a learning rate too high, the model does not have time to learn anything. This is the case in the left picture. The line is moving up and down, meaning the model predicts with pure guess the outcome. The picture on the right shows that the loss is decreasing over iteration until the curve got flatten, meaning the model found a solution.

TensorBoard is a great tool to visualize such metrics and highlight potential issues. The neural network can take hours to weeks before they find a solution. TensorBoard updates the metrics very often. In this case, you don’t need to wait until the end to see if the model trains correctly. You can open TensorBoard to check how the training is going and make the appropriate change if necessary.

How to use TensorBoard?

In this tutorial, you will learn how to open TensorBoard from the terminal for MacOS and the Command line TensorBoard for Windows.

The code will be explained in a future tutorial, the focus here is on TensorBoard.

First, you need to import the libraries you will use during the training## Import the library import tensorflow as tf import numpy as np

You create the data. It is an array of 10000 rows and 5 columnsX_train = (np.random.sample((10000,5))) y_train = (np.random.sample((10000,1))) X_train.shape

Output

(10000, 5)

The codes below transform the data and create the model.

Note that the learning rate is equal to 0.1. If you change this rate to a higher value, the model will not find a solution. This is what happened on the left side of the above picture.

During most of the TensorFlow tutorials, you will use TensorFlow estimator. This is TensorFlow API that contains all the mathematical computations.

To create the log files, you need to specify the path. This is done with the argument model_dir.

In the TensorBoard example below, you store the model inside the working directory, i.e., where you store the notebook or python file. Inside this path, TensorFlow will create a folder called train with a child folder name linreg.feature_columns = [ tf.feature_column.numeric_column(‘x’, shape=X_train.shape[1:])] DNN_reg = tf.estimator.DNNRegressor(feature_columns=feature_columns, # Indicate where to store the log file model_dir=’train/linreg’, hidden_units=[500, 300], optimizer=tf.train.ProximalAdagradOptimizer( learning_rate=0.1, l1_regularization_strength=0.001 ) )

Output

INFO:tensorflow:Using default config. INFO:tensorflow:Using config: {‘_model_dir’: ‘train/linreg’, ‘_tf_random_seed’: None, ‘_save_summary_steps’: 100, ‘_save_checkpoints_steps’: None, ‘_save_checkpoints_secs’: 600, ‘_session_config’: None, ‘_keep_checkpoint_max’: 5, ‘_keep_checkpoint_every_n_hours’: 10000, ‘_log_step_count_steps’: 100, ‘_train_distribute’: None, ‘_service’: None, ‘_cluster_spec’: <tensorflow.python.training.server_lib.ClusterSpec object at 0x1818e63828>, ‘_task_type’: ‘worker’, ‘_task_id’: 0, ‘_global_id_in_cluster’: 0, ‘_master’: ”, ‘_evaluation_master’: ”, ‘_is_chief’: True, ‘_num_ps_replicas’: 0, ‘_num_worker_replicas’: 1}

The last step of this TensorFlow visualize graph example consists to train the model. During the training, TensorFlow writes information in the model directory.# Train the estimator train_input = tf.estimator.inputs.numpy_input_fn( x={“x”: X_train}, y=y_train, shuffle=False,num_epochs=None) DNN_reg.train(train_input,steps=3000)

Output

INFO:tensorflow:Calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Saving checkpoints for 1 into train/linreg/model.ckpt. INFO:tensorflow:loss = 40.060104, step = 1 INFO:tensorflow:global_step/sec: 197.061 INFO:tensorflow:loss = 10.62989, step = 101 (0.508 sec) INFO:tensorflow:global_step/sec: 172.487 INFO:tensorflow:loss = 11.255318, step = 201 (0.584 sec) INFO:tensorflow:global_step/sec: 193.295 INFO:tensorflow:loss = 10.604872, step = 301 (0.513 sec) INFO:tensorflow:global_step/sec: 175.378 INFO:tensorflow:loss = 10.090343, step = 401 (0.572 sec) INFO:tensorflow:global_step/sec: 209.737 INFO:tensorflow:loss = 10.057928, step = 501 (0.476 sec) INFO:tensorflow:global_step/sec: 171.646 INFO:tensorflow:loss = 10.460144, step = 601 (0.583 sec) INFO:tensorflow:global_step/sec: 192.269 INFO:tensorflow:loss = 10.529617, step = 701 (0.519 sec) INFO:tensorflow:global_step/sec: 198.264 INFO:tensorflow:loss = 9.100082, step = 801 (0.504 sec) INFO:tensorflow:global_step/sec: 226.842 INFO:tensorflow:loss = 10.485607, step = 901 (0.441 sec) INFO:tensorflow:global_step/sec: 152.929 INFO:tensorflow:loss = 10.052481, step = 1001 (0.655 sec) INFO:tensorflow:global_step/sec: 166.745 INFO:tensorflow:loss = 11.320213, step = 1101 (0.600 sec) INFO:tensorflow:global_step/sec: 161.854 INFO:tensorflow:loss = 9.603306, step = 1201 (0.619 sec) INFO:tensorflow:global_step/sec: 179.074 INFO:tensorflow:loss = 11.110269, step = 1301 (0.556 sec) INFO:tensorflow:global_step/sec: 202.776 INFO:tensorflow:loss = 11.929443, step = 1401 (0.494 sec) INFO:tensorflow:global_step/sec: 144.161 INFO:tensorflow:loss = 11.951693, step = 1501 (0.694 sec) INFO:tensorflow:global_step/sec: 154.144 INFO:tensorflow:loss = 8.620987, step = 1601 (0.649 sec) INFO:tensorflow:global_step/sec: 151.094 INFO:tensorflow:loss = 10.666125, step = 1701 (0.663 sec) INFO:tensorflow:global_step/sec: 193.644 INFO:tensorflow:loss = 11.0349865, step = 1801 (0.516 sec) INFO:tensorflow:global_step/sec: 189.707 INFO:tensorflow:loss = 9.860596, step = 1901 (0.526 sec) INFO:tensorflow:global_step/sec: 176.423 INFO:tensorflow:loss = 10.695, step = 2001 (0.567 sec) INFO:tensorflow:global_step/sec: 213.066 INFO:tensorflow:loss = 10.426752, step = 2101 (0.471 sec) INFO:tensorflow:global_step/sec: 220.975 INFO:tensorflow:loss = 10.594796, step = 2201 (0.452 sec) INFO:tensorflow:global_step/sec: 219.289 INFO:tensorflow:loss = 10.4212265, step = 2301 (0.456 sec) INFO:tensorflow:global_step/sec: 215.123 INFO:tensorflow:loss = 9.668612, step = 2401 (0.465 sec) INFO:tensorflow:global_step/sec: 175.65 INFO:tensorflow:loss = 10.009649, step = 2501 (0.569 sec) INFO:tensorflow:global_step/sec: 206.962 INFO:tensorflow:loss = 10.477722, step = 2601 (0.483 sec) INFO:tensorflow:global_step/sec: 229.627 INFO:tensorflow:loss = 9.877638, step = 2701 (0.435 sec) INFO:tensorflow:global_step/sec: 195.792 INFO:tensorflow:loss = 10.274586, step = 2801 (0.512 sec) INFO:tensorflow:global_step/sec: 176.803 INFO:tensorflow:loss = 10.061047, step = 2901 (0.566 sec) INFO:tensorflow:Saving checkpoints for 3000 into train/linreg/model.ckpt. INFO:tensorflow:Loss for final step: 10.73032. <tensorflow.python.estimator.canned.dnn.DNNRegressor at 0x1818e63630>

For MacOS user

For Windows user

You can see this information in the PyTorch TensorBoard.

Now that you have the log events written, you can open Tensorboard. Tensorboard Keras runs on port 6006 (Jupyter runs on port 8888). You can use the Terminal for MacOs users or Anaconda prompt for Windows user.

For MacOS user

# Different for you cd /Users/Guru99/tuto_TF source activate hello-tf!

The notebook is stored in the path /Users/Guru99/tuto_TF

For Windows users

cd C:\Users\Admin\Anaconda3 activate hello-tf

The notebook is stored in the path C:\Users\Admin\Anaconda3

To launch Tensorboard, you can use this code

For MacOS user

tensorboard –logdir=./train/linreg

For Windows users

tensorboard –logdir=.\train\linreg

Tensorboard is located in this URL: http://localhost:6006

It could also be located at the following location.

Copy and paste the URL into your favorite browser. You should see this:

Note that, we will learn how to read the graph in the tutorial dedicated to the deep learning.

If you see something like this:

It means Tensorboard cannot find the log file. Make sure you point the cd to the right path or double check if the log event has been creating. If not, re-run the code.

If you want to close TensorBoard Press CTRL+C

Hat Tip: Check your anaconda prompt for the current working directory,