Large Scale Data Analysis Using Deep Learning (Prof. U Kang) Introduction to TensorFlow 2017.04.17 Beunguk Ahn ( beunguk.ahn@gmail.com) 1
What is TensorFlow? Consturction Phase Execution Phase Examples Gotchas and Tips Contents 2
What is TensorFlow? Consturction Phase Execution Phase Examples Gotchas and Tips Contents 3
Prerequisites You already know how to use Python You already installed TensorFlow 4
TensorFlow Open source so ware library for numerical computation using data flow graphs. Simply, it is one of widely used frameworks for deep learning programs. 5
TensorFlow Open source so ware library for numerical computation using data flow graphs. Simply it is one of widely used frameworks for deep learning programs. 6
Role of TensorFlow Machine learning flow in general 7
Why use TensorFlow? Large community engagement (Google) Multiple GPU Support Data and Model Parallelism Python + Numpy Tensorboard However, you don't have to use TensorFlow. Other frameworks like Theano, Caffe, Torch, etc are also good options. 8
Tensor Tensor is simply multi dimension array 1-D Tensor : vector 2-D Tensor : matrix 3-D Tensor : tensor 4-D Tensor : tensor... 1Dtensor = [ 1, 2] # vector 2Dtensor = [[ 1, 2], # matrix [ 3,4]] 3Dtensor = [[[ 1, 2], # tensor [ 3, 4]], [[ 5, 6], [ 7, 8]], [[ 9,10], [11,12]]] 9
Shape of Tensor Shape of tensor is expressed as tuple import tensorflow as tf scalar = tf.constant(1) # () zero = tf.constant([]) # (0,) vector = tf.constant([1,2,3,4]) # (4,) matrix = tf.constant([[1,2],[3,4]]) # (2, 2) tensor = tf.constant([[[1],[ 3]], # (3, 2, 1) [[5],[ 7]], [[9],[11]]]) print '\n'.join( map(str, [scalar, zero, vector, matrix, tensor])) >>> Tensor("Const:0", shape=(1,), dtype=int32) >>> Tensor("Const_1:0", shape=(0,), dtype=float32) >>> Tensor("Const_2:0", shape=(4,), dtype=int32) >>> Tensor("Const_3:0", shape=(2, 2), dtype=int32) >>> Tensor("Const_4:0", shape=(3, 2, 1), dtype=int32) 10
Data Type of Tensor What kind of data contained in tensor import tensorflow as tf int32 = tf.constant([[1,2],[3,4]]) # int32 float32 = tf.constant([[1.0, 2], [3, 4.0]]) # float32 float64 = tf.constant([[1.0, 2], [3, 4.0]], # float64 dtype=tf.float64) _string = tf.constant([["a", "B"], ["C", "D"]], # string dtype=tf.string) print '\n'.join( map(str, [int32, float32, float64, _string])) >>> Tensor("Const:0", shape=(2, 2), dtype=int32) >>> Tensor("Const_1:0", shape=(2, 2), dtype=float32) >>> Tensor("Const_2:0", shape=(2, 2), dtype=float64) >>> Tensor("Const_3:0", shape=(2, 2), dtype=string) Also float(32,64), [u]int(8,16,32,64), bool, complex(64,128) are also available 11
TensorFlow program in two parts Construction Phase Build a graph Declare Variables, Placeholders, Operations, Optimizers Execution Phase Initialize model and run Start session and iterate through the graph 12
What is TensorFlow? Consturction Phase Execution Phase Examples Gotchas and Tips Contents 13
Variable Stores a scalar or tensor Used as an input for Operations Requires initial value for future use At constuction phase, variables are "just" defined. Value will be assigned a er Session initialized You will get an idea through slides 14
Variable import tensorflow as tf # Define variable, yet initialized w = tf.variable(tf.random_normal([1, 3]), name="weight") with tf.session() as sess: w_value = sess.run(w) # get value for var 'w' print '\n'.join(map(str, ([w, w_value]))) "tensorflow.python.framework.errors_impl. FailedPreconditionError: Attempting to use uninitialized value weight" 15
Variable import tensorflow as tf # Define variable, yet initialized w = tf.variable(tf.random_normal([1, 3]), name="weight") # Define init operation init_op = tf.global_variables_initializer() with tf.session() as sess: sess.run(init_op) # initialize all declared variables w_value = sess.run(w) # get value for var 'w' print '\n'.join(map(str, ([w, w_value]))) >>> Tensor("weight/read:0", shape=(2, 3), dtype=float32) >>> [[-0.50282216-1.94712341-0.47356459]] 16
Variable import tensorflow as tf # Define variable, yet initialized w = tf.variable(tf.random_normal([1, 3]), name="weight") # Define init operation init_op = tf.global_variables_initializer() with tf.session() as sess: sess.run(init_op) # initialize all declared variables w_value = sess.run(w); x_value = sess.run(w) sess.run(init_op) # re-initialize y_value = sess.run(w) print '\n'.join(map(str, ([w_value, x_value, y_value]))) >>> [[-0.50282216-1.94712341-0.47356459]] >>> [[-0.50282216-1.94712341-0.47356459]] # same >>> [[ 1.1028192 0.25910807 2.37004709]] # changed 17
Placeholder Placeholder for tensor Must be fed with data on execution for operation that takes the placeholder as operand 18
Placeholder import tensorflow as tf x = [[1, 2]] y = [[1], [3]] pla = tf.placeholder(tf.int32, (1,2)) matmul = tf.matmul(pla, y) with tf.session() as sess: A = sess.run(matmul) print '\n'.join(map(str, ([pla, A]))) >>> "You must feed a value for placeholder tensor 'Placeholder' with dtype int32 and shape [1,2]" 19
Placeholder import tensorflow as tf x = [[1, 2]] y = [[1], [3]] pla = tf.placeholder(tf.int32, (1,2)) matmul = tf.matmul(pla, y) with tf.session() as sess: A = sess.run(matmul, feed_dict={pla : x}) print '\n'.join(map(str, ([pla, A]))) >>> Tensor("Placeholder:0", shape=(2, 3), dtype=int32) >>> [[7]] 20
Shape of Placeholder Shape could be partially known import tensorflow as tf p1 = tf.placeholder(tf.float32, [1, None]) p2 = tf.placeholder(tf.float32, [None, 2]) p3 = tf.placeholder(tf.float32, [None, None]) ops = [tf.multiply(x, 10) for x in [p1, p2, p3]] mat1 = [[1, 2]] mat2 = [[3, 4], [5, 6]] mat3 = [[1], [2]] with tf.session() as sess: print '\n'.join(map(str, sess.run(ops, feed_dict={ p1:mat1, p2:mat2, p3:mat3}))) >>> [[ 10. 20.]] >>> [[ 30. 40.] [ 50. 60.]] >>> [[ 10.] [ 20.]] 21
Shape of Placeholder Useful when you feed input/output to your model without giving the size of batch import tensorflow as tf # Placeholder MNIST # image size 28x28 x = tf.placeholder(tf.float32, [None, 784]) # 10 classes [0..9] y_ = tf.placeholder(tf.float32, [None, 10]) # Softmax regression Wx = b W = tf.variable(tf.zeros([784, 10])) b = tf.variable(tf.zeros([10])) y = tf.nn.softmax(tf.matmul(x, W) + b)... 22
Operation Operation is graph node that performs computation on tensors Computation should be run on Session It takes inputs and generates outputs(optional) Usually, it ends up with prediction, cost function, or updating variables We have already seen some operations through the slides 23
Operation Linear Regression (Mean square error) 1 xw + b = y, L = ( x W i=1 i y i ) 2 2m import tensorflow as tf, numpy as np # tf Graph Input X = tf.placeholder("float") Y = tf.placeholder("float") m # Set model weights W = tf.variable(np.random.randn(), name="weight") b = tf.variable(np.random.randn(), name="bias") # Construct a linear model pred = tf.add(tf.multiply(x, W), b) # Mean square error cost = tf.reduce_sum(tf.pow(pred-y, 2))/(2*n_samples) 24
Operation It is better to read through documentation so that you don't have to reinvent the wheel Math matmul, reduce_sum, argmax, norm... Assign assign, assign_add, assign_sub... Tensor transform slice, split, stack, concat... 25
Optimizer Provides methods to compute gradients for a loss and apply gradients to variables Getting cost function as input and Calcualte gradients Process gradients Apply gradients "minimize" method on Optmizers do all 3 steps Many optimizers are alreafy implemented Gradient Descent Optimizer, Adagrad Optimizer, RMSProp Optmizer... 26
Optimizer import tensorflow as tf, numpy as np... # Construct a linear model pred = tf.add(tf.multiply(x, W), b) # Mean square error cost = tf.reduce_sum(tf.pow(pred-y, 2))/(2*n_samples) # Gradient descent optimizer = tf.train\.gradientdescentoptimizer(learning_rate)\.minimize(cost) with tf.session() as sess: for epoch in range(training_epochs): for (x, y) in zip(train_x, train_y): # SGD m'=1 sess.run(optimizer, feed_dict={x: x, Y: y}) # update weights 27
What is TensorFlow? Consturction Phase Execution Phase Examples Gotchas and Tips Contents 28
Session Session launchs the graph, initalize them, and run operations as you defined We have seen sessions through the slide, and I hope you got the idea of what it is 29
Session Your data will be lost when you close the session import tensorflow as tf var = tf.variable(tf.random_normal((1, 3))) init = tf.global_variables_initializer() with tf.session() as sess: sess.run(init) print sess.run(var) # Fetch train data # Calcuate cost # Optimize # Do Test # Display or Save print var.eval() "No default session is registered. Use `with sess.as_default()` or pass an explicit session to..." 30
What is TensorFlow? Consturction Phase Execution Phase Examples Gotchas and Tips Contents 31
MNIST Famous example for deep learning Data: Hand-writing digit[0-9] image data (28*28) Each image will be treated as 784 length of vector Task: Classify what digit in each image 10 classes Model: So max regression model Also use 3 hidden layers 32
x = set of images y = set of labels laye r 1 = W 1 x + b 1 MNIST laye r 2 = W 2 laye r 1 + b 2 pred = W 3 laye r 2 + b 3 m 1 cost = y log(softmax(pred )) m i=1 i y i 33
Hyper parameters # Parameters learning_rate = 0.001 training_epochs = 15 batch_size = 100 display_step = 1 # Network Parameters n_hidden_1 = 256 # 1st layer number of features n_hidden_2 = 256 # 2nd layer number of features n_input = 784 # MNIST data input (img shape: 28*28) n_classes = 10 # MNIST total classes (0-9 digits) Reference: TensorFlow Exmaples 34
Placeholder & Variables import tensorflow as tf # tf Graph input x = tf.placeholder("float", [None, n_input]) # input y = tf.placeholder("float", [None, n_classes]) # output # Store layers weight & bias h1 = tf.variable(tf.random_normal([n_input, n_hidden_1])) h2 = tf.variable(tf.random_normal([n_hidden_1, n_hidden_2])) out = tf.variable(tf.random_normal([n_hidden_2, n_classes])) b1 = tf.variable(tf.random_normal([n_hidden_1])) b2 = tf.variable(tf.random_normal([n_hidden_2])) b3 = tf.variable(tf.random_normal([n_classes])) 35
36
Build a model # Hidden layer 1 with RELU activation layer_1 = tf.add(tf.matmul(x, h1), b1) layer_1 = tf.nn.relu(layer_1) # Hidden layer 2 with RELU activation layer_2 = tf.add(tf.matmul(layer_1, h2), b2) layer_2 = tf.nn.relu(layer_2) # Output layer with linear activation pred = tf.matmul(layer_2, out) + b3 37
38
Optimizer # Define loss and optimizer y_hat_softmax = tf.nn.softmax(pred) cost = tf.reduce_mean( -tf.reduce_sum(y * tf.log(y_hat_softmax), [1])) optimizer = tf.train.adamoptimizer( learning_rate=learning_rate)\.minimize(cost) # Initializing the variables init = tf.global_variables_initializer() 39
40
Run & Learn # Launch the graph with tf.session() as sess: sess.run(init) # Training cycle for epoch in range(training_epochs): avg_cost = 0. total_batch = int(mnist.train.num_examples/batch_size) # Loop over all batches for i in range(total_batch): batch_x, batch_y = mnist.train.next_batch(batch_size) # Run optimization op (backprop) and cost op (to get loss value) _, c = sess.run([optimizer, cost], feed_dict={x: batch_x, y: batch_y}) # Compute average loss avg_cost += c / total_batch # Display logs per epoch step if epoch % display_step == 0: print "Epoch:", '%04d' % (epoch+1), "cost=", \ "{:.9f}".format(avg_cost) print "Optimization Finished!" # Test model correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1)) # Calculate accuracy accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) print "Accuracy:", accuracy.eval({x: mnist.test.images, y: mnist.test.labels}) 41
Result >>>... >>> Epoch: 0010 cost= 3.325720582 >>> Epoch: 0011 cost= 2.454958990 >>> Epoch: 0012 cost= 1.861004718 >>> Epoch: 0013 cost= 1.412745127 >>> Epoch: 0014 cost= 1.158697252 >>> Epoch: 0015 cost= 0.943136121 >>> Optimization Finished! >>> Accuracy: 0.9485 42
What is TensorFlow? Consturction Phase Execution Phase Examples Gotchas and Tips Contents 43
Gotchas & Tips Don't forget to put value to feed_dict import tensorflow as tf pla = tf.placeholder(tf.int32, [2,2]) x = tf.variable([[1,2],[3,4]]) y = tf.variable([[3,4],[1,2]]) init = tf.global_variables_initializer() matmul = tf.matmul(pla, y) with tf.session() as sess: sess.run(init) f_dict = { pla: x } # x is actually tensor "object" result = sess.run(matmul, feed_dict=f_dict) print result "ValueError: setting an array element with a sequence." 44
Gotchas & Tips Don't forget to put value to feed_dict import tensorflow as tf pla = tf.placeholder(tf.int32, [2,2]) x = tf.variable([[1,2],[3,4]]) y = tf.variable([[3,4],[1,2]]) init = tf.global_variables_initializer() matmul = tf.matmul(pla, y) with tf.session() as sess: sess.run(init) x_v = sess.run(x) f_dict = { pla: x_v } # x_v is now "array(2,2)" result = sess.run(matmul, feed_dict=f_dict) print result >>> [[ 5 8] [13 20]] 45
Gotchas & Tips Better not to make Variable/Placeholder/Operation inside epoch import tensorflow as tf with tf.session() as sess: for epoch in range(100): # It will make session overflow! v = tf.variable([[1,2]]) op = tf.add(1,2) pla = tf.placeholder(tf.int32, [1,2]) print '\n'.join(map(str, [v, op, pla])) >>>... >>> Tensor("Variable_99/read:0", shape=(1, 2), dtype=int32) >>> Tensor("Add_99:0", shape=(), dtype=int32) >>> Tensor("Placeholder_99:0", shape=(1, 2), dtype=int32) 46
Useful References TensorFlow-Examples DCGAN-tensorflow Style Transfer in TensorFlow Word2vec in TensorFlow 47
Q & A 48