Mahmoud Abas, Full-Stack Software Engineer & AI

Neural Networks - Introduction to TensorFlow 2.x

In the print dialog, choose "Save as PDF" as the destination.

Neural Networks - Lesson 2

Introduction to TensorFlow 2.x

Core data structures, math operations, computation graphs, and autodiff

Why TensorFlow 2.x?

Open-source deep learning framework by Google; eager execution by default
Eager Execution: operations run immediately, easy to debug
Keras as the primary API: tf.keras is the recommended high-level API
Supports the full ML workflow: preprocessing, training, evaluation, deployment
Runs on CPU, GPU, TPU, and mobile devices; supports distributed training

python

import tensorflow as tf
print("TensorFlow Version:", tf.__version__)
print("Is GPU available?:", tf.config.list_physical_devices('GPU'))

Core Data Structures

tf.Tensor - immutable n-dimensional array (the fundamental structure)
tf.Variable - mutable tensor, used to store trainable parameters
tf.TensorArray - dynamically sized array of tensors (used in RNNs)
tf.RaggedTensor - tensors with variable-length rows
tf.SparseTensor - efficient storage for mostly-zero data

tf.Tensor: Ranks and Immutability

A scalar is a rank-0 tensor (no axes); a vector is rank-1 (one axis); a matrix is rank-2 (two axes)
Tensors are immutable - you cannot update contents, only create a new tensor

python

rank_0_tensor = tf.constant(4)
# tf.Tensor(4, shape=(), dtype=int32)

rank_1_tensor = tf.constant([2.0, 3.0, 4.0])
# tf.Tensor([2. 3. 4.], shape=(3,), dtype=float32)

rank_2_tensor = tf.constant([[1, 2], [3, 4], [5, 6]], dtype=tf.float16)
# shape=(3, 2), dtype=float16

Convert to NumPy with np.array(tensor) or tensor.numpy()

Tensor Attributes, Indexing, Slicing

The most important attributes: .shape (size per axis) and .dtype (element type)
Tensors support NumPy-style indexing and slicing

python

tensor = tf.constant([[1, 2], [3, 4]], dtype=tf.float32)
print("Shape:", tensor.shape)      # (2, 2)
print("Data Type:", tensor.dtype)  # float32

tensor[0, 0]   # single element -> 1.0
tensor[:, 0]   # all rows, col 0 -> [1. 3.]

tf.Variable, RaggedTensor, SparseTensor

tf.Variable is mutable - used to store and update model weights during training

python

var = tf.Variable([[5.0, 6.0], [7.0, 8.0]])
var.assign([[1, 2], [3, 4]])
var.assign_add([[1, 1], [1, 1]])
# Incremented Value: [[2. 3.] [4. 5.]]

tf.RaggedTensor handles variable-length rows (e.g. sentences of different lengths)
tf.SparseTensor efficiently represents mostly-zero data; convert with tf.sparse.to_dense()

Numerical APIs: Basic Math

TensorFlow provides math, linear algebra, and random-number utilities for model building

python

x = tf.constant([2.0, 4.0, 6.0])
y = tf.constant([1.0, 2.0, 3.0])

tf.add(x, y)       # [3. 6. 9.]
tf.subtract(x, y)  # [1. 2. 3.]
tf.multiply(x, y)  # [2. 8. 18.]
tf.divide(x, y)    # [2. 2. 2.]

Linear Algebra Operations

tf.matmul for matrix multiplication - the backbone of neural network layers
Also available: tf.linalg.det, tf.linalg.inv, tf.linalg.eig, tf.linalg.svd

python

A = tf.constant([[2.0, 3.0], [1.0, 4.0]])
B = tf.constant([[1.0, 2.0], [3.0, 4.0]])

tf.linalg.matmul(A, B)
# [[11. 16.]
#  [13. 18.]]

Random Number Generation and reduce_mean

tf.random.normal, tf.random.uniform, tf.random.shuffle, tf.random.set_seed for reproducibility
tf.reduce_mean averages elements, optionally along a specific axis

python

tf.random.normal([3, 3], mean=0.0, stddev=1.0)
tf.random.uniform([2, 2], minval=0, maxval=10)
tf.random.set_seed(42)

x = tf.constant([[1, 2, 3], [4, 5, 6]], dtype=tf.float32)
tf.reduce_mean(x)          # 3.5 (all elements)
tf.reduce_mean(x, axis=0)  # [2.5 3.5 4.5] (per column)
tf.reduce_mean(x, axis=1)  # [2. 5.]        (per row)

Computation Graphs: Eager vs tf.function

TensorFlow 2.x executes eagerly by default - operations run immediately, easy to debug
@tf.function traces a Python function into a static graph for faster, optimized execution
Trade-off: graph mode is faster but has constraints (e.g. control flow must be TF-compatible)

Python function

eager, runs immediately

@tf.function traces it

first call

Static graph

Faster execution

every later call

python

@tf.function
def computation_graph(x1, x2):
    add_op = x1 + x2
    mul_op = x1 * x2
    return add_op - mul_op

x1 = tf.constant(7, dtype=tf.float32)
x2 = tf.constant(3, dtype=tf.float32)
computation_graph(x1, x2)  # -11

Automatic Differentiation: tf.GradientTape

TensorFlow computes gradients via reverse-mode autodiff
Record operations inside a tf.GradientTape() context, then call .gradient()

y = x³

Worked example: if y = x^3 and x = 2.0, what is dy/dx?

dydx = 3x² = 12.0

python

x = tf.Variable(2.0)
with tf.GradientTape() as tape:
    y = x ** 3  # y = x^3

dy_dx = tape.gradient(y, x)
# dy/dx = 3*x^2 = 12.0

GradientTape with Multiple Variables

Neural networks track gradients for many trainable variables at once

python

w = tf.Variable(1.0)
b = tf.Variable(2.0)
x_vals = tf.constant([1.0, 2.0, 3.0])

with tf.GradientTape() as tape:
    y_vals = w * x_vals + b
    loss = tf.reduce_mean((y_vals - tf.constant([2.0, 4.0, 6.0]))**2)

gradients = tape.gradient(loss, [w, b])
# dLoss/dw: -1.333, dLoss/db: 0.0

Key Takeaways

tf.Tensor is immutable; tf.Variable is mutable and used for trainable parameters
Tensors have ranks (scalar, vector, matrix, ...) and are described by .shape and .dtype
TensorFlow provides math, linear algebra (tf.matmul), and random-number APIs built for ML
Eager execution runs ops immediately; @tf.function compiles a faster static graph
tf.GradientTape() records operations and computes gradients automatically via .gradient()
Autodiff via GradientTape is the mechanism that powers training in every neural network

Open this lesson

Mahmoud AbasIntroduction to TensorFlow 2.x

rank_0_tensor = tf.constant(4) # tf.Tensor(4, shape=(), dtype=int32) rank_1_tensor = tf.constant([2.0, 3.0, 4.0]) # tf.Tensor([2. 3. 4.], shape=(3,), dtype=float32) rank_2_tensor = tf.constant([[1, 2], [3, 4], [5, 6]], dtype=tf.float16) # shape=(3, 2), dtype=float16

tensor = tf.constant([[1, 2], [3, 4]], dtype=tf.float32) print("Shape:", tensor.shape) # (2, 2) print("Data Type:", tensor.dtype) # float32 tensor[0, 0] # single element -> 1.0 tensor[:, 0] # all rows, col 0 -> [1. 3.]

tf.random.normal([3, 3], mean=0.0, stddev=1.0) tf.random.uniform([2, 2], minval=0, maxval=10) tf.random.set_seed(42) x = tf.constant([[1, 2, 3], [4, 5, 6]], dtype=tf.float32) tf.reduce_mean(x) # 3.5 (all elements) tf.reduce_mean(x, axis=0) # [2.5 3.5 4.5] (per column) tf.reduce_mean(x, axis=1) # [2. 5.] (per row)

@tf.function def computation_graph(x1, x2): add_op = x1 + x2 mul_op = x1 * x2 return add_op - mul_op x1 = tf.constant(7, dtype=tf.float32) x2 = tf.constant(3, dtype=tf.float32) computation_graph(x1, x2) # -11

w = tf.Variable(1.0) b = tf.Variable(2.0) x_vals = tf.constant([1.0, 2.0, 3.0]) with tf.GradientTape() as tape: y_vals = w * x_vals + b loss = tf.reduce_mean((y_vals - tf.constant([2.0, 4.0, 6.0]))**2) gradients = tape.gradient(loss, [w, b]) # dLoss/dw: -1.333, dLoss/db: 0.0