Neural Networks

Neural Networks - Setup

In the print dialog, choose "Save as PDF" as the destination.

Neural Networks - Lesson 4

Neurons as Logic Gates

Using the sigmoid neuron to implement, learn, and outgrow basic logic gates

Setup

python

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
import itertools as it

Standard imports for this notebook

Goal: use neurons to implement the most primitive logic computations, logic gates

The Sigmoid Function

σ = 11 + e^-x

σ ranges from 0 to 1: negative x pushes it toward 0, positive x pushes it toward 1, and σ(0) = 0.5

python

def sigmoid(x):
    """Sigmoid function"""
    return 1.0 / (1.0 + np.exp(-x))

xs = np.linspace(-10, 10, num=100, dtype=np.float32)
plt.plot(xs, sigmoid(xs))
plt.plot(0, .5, 'ro')

OR as a Neuron

Logic gate: Takes two boolean inputs and returns 0 or 1 by a fixed rule.

z = w₁ x₁ + w₂ x₂ + b

A sigmoid neuron computes z, then squashes z through the sigmoid function
To match OR: when both inputs are 0, z must be very negative (a negative b); when either input is 1, z must swing very positive

x1	x2	OR
0	0	0
0	1	1
1	0	1
1	1	1

Working weights: w1 = 20, w2 = 20, b = -10

Testing the OR Neuron

Helper functions: logic_gate builds a two-input sigmoid neuron; test_gate prints its truth table

python

def logic_gate(w1, w2, b, x1, x2):
  return sigmoid(w1 * x1 + w2 * x2 + b)

def test_gate(w1, w2, b):
  for x1 in range(2):
    for x2 in range(2):
      print("{}, {}: {}".format(x1, x2, np.round(logic_gate(w1, w2, b, x1, x2))))

or_gate = test_gate(20, 20, -10)
# 0, 0: 0.0
# 0, 1: 1.0
# 1, 0: 1.0
# 1, 1: 1.0

Matches the OR truth table exactly

Exercise

AND, NOR, NAND Gates

Same approach, different weights and bias, derive them logically rather than by guessing
AND: output 1 only when both inputs are 1
NOR: output 1 only when both inputs are 0 (the opposite of OR)
NAND: output 1 unless both inputs are 1 (the opposite of AND)

python

# Fill in w1, w2, b so the truth table matches:
# and_gate = test_gate(...)
# nor_gate = test_gate(...)
# nand_gate = test_gate(...)

Each gate is just a different point in (w1, w2, b) space for the same sigmoid neuron

Learning a Logic Gate with TensorFlow

Instead of hand-deriving weights, train a tf.Module model on the truth table as data
Weights and bias are initialized randomly on first call, then learned via gradient descent

python

class LogicGate(tf.Module):
    def __init__(self):
        super().__init__()
        self.built = False

    def __call__(self, x, train=True):
        if not self.built:
            input_dim = x.shape[-1]
            self.w = tf.Variable(tf.random.normal([input_dim, 1]))
            self.b = tf.Variable(tf.zeros([1]))
            self.built = True
        z = tf.add(tf.matmul(x, self.w), self.b)
        return tf.sigmoid(z)

Training and Accuracy Functions

MSE loss between predicted and true gate outputs
Standard gradient-descent loop: forward pass, compute gradients, update variables

python

def compute_loss(y_pred, y_true):
    return tf.reduce_mean(tf.square(y_pred - y_true))

def train_model(model, x_train, y_train, learning_rate=0.5, epochs=5000):
  for epoch in range(epochs):
    with tf.GradientTape() as tape:
      y_pred = model(x_train)
      loss = compute_loss(y_pred, y_train)
    grads = tape.gradient(loss, model.variables)
    for g, v in zip(grads, model.variables):
      v.assign_sub(learning_rate * g)

def compute_accuracy(model, x, y_true):
    y_pred = tf.round(model(x, train=False))
    correct = tf.equal(y_pred, y_true)
    return tf.reduce_mean(tf.cast(correct, tf.float32)).numpy()

Training the AND Gate

Train on the four rows of the AND truth table as (x1, x2) -> y pairs

python

and_table = np.array([[0, 0, 0], [1, 0, 0], [0, 1, 0], [1, 1, 1]], dtype=np.float32)
x_train = and_table[:, :2]
y_train = and_table[:, 2:]

model = LogicGate()
train_model(model, x_train, y_train)

text

Epoch 0, Loss: 0.3312, Accuracy: 0.5000
Epoch 1000, Loss: 0.0126, Accuracy: 1.0000
Epoch 4000, Loss: 0.0027, Accuracy: 1.0000

Learned weight for w1: 5.733
Learned weight for w2: 5.733
Learned bias: -8.692

The model reaches 100% accuracy and learns weights structurally similar to the hand-derived OR solution

The Limits of a Single Neuron

XOR

XOR outputs 1 only when inputs differ, the basis of computation (half-adders)

x1	x2	XOR
0	0	0
0	1	1
1	0	1
1	1	0

Composing Neurons to Build XOR

Chain multiple gates: XOR(A, B) = OR(A, B) AND NAND(A, B)
Two neurons (OR, NAND) feed a third neuron (AND), a two-layer network

Inputs A, B

gate 1

NAND

gate 2

AND

gate 3

XOR

output

python

# def xor_gate(a, b):
#     c = or_gate(a, b)
#     d = nand_gate(a, b)
#     return and_gate(c, d)

Chaining neurons composes more complex functions than any single neuron can represent

Key Takeaways

A sigmoid neuron computes σ(w1*x1 + w2*x2 + b), the building block of logic-gate-like behavior
Hand-derived weights can implement OR, AND, NOR, NAND directly by reasoning about the sigmoid's saturation points
Gradient descent can learn these same weights automatically from truth-table data using tf.GradientTape
A single neuron is a linear classifier, it cannot represent XOR (not linearly separable)
XOR requires composing multiple neurons and layers: XOR = OR AND NAND
This limitation of single neurons motivates multilayer networks, covered next

Open this lesson

Mahmoud AbasSetup

def logic_gate(w1, w2, b, x1, x2): return sigmoid(w1 * x1 + w2 * x2 + b) def test_gate(w1, w2, b): for x1 in range(2): for x2 in range(2): print("{}, {}: {}".format(x1, x2, np.round(logic_gate(w1, w2, b, x1, x2)))) or_gate = test_gate(20, 20, -10) # 0, 0: 0.0 # 0, 1: 1.0 # 1, 0: 1.0 # 1, 1: 1.0

class LogicGate(tf.Module): def __init__(self): super().__init__() self.built = False def __call__(self, x, train=True): if not self.built: input_dim = x.shape[-1] self.w = tf.Variable(tf.random.normal([input_dim, 1])) self.b = tf.Variable(tf.zeros([1])) self.built = True z = tf.add(tf.matmul(x, self.w), self.b) return tf.sigmoid(z)

def compute_loss(y_pred, y_true): return tf.reduce_mean(tf.square(y_pred - y_true)) def train_model(model, x_train, y_train, learning_rate=0.5, epochs=5000): for epoch in range(epochs): with tf.GradientTape() as tape: y_pred = model(x_train) loss = compute_loss(y_pred, y_train) grads = tape.gradient(loss, model.variables) for g, v in zip(grads, model.variables): v.assign_sub(learning_rate * g) def compute_accuracy(model, x, y_true): y_pred = tf.round(model(x, train=False)) correct = tf.equal(y_pred, y_true) return tf.reduce_mean(tf.cast(correct, tf.float32)).numpy()

Epoch 0, Loss: 0.3312, Accuracy: 0.5000 Epoch 1000, Loss: 0.0126, Accuracy: 1.0000 Epoch 4000, Loss: 0.0027, Accuracy: 1.0000 Learned weight for w1: 5.733 Learned weight for w2: 5.733 Learned bias: -8.692

XOR