Let’s continue from day 14 which explained the Keras API and compare them
Part 1: Understanding Sequential vs. Functional API in Keras with a Simple Example
When building neural networks in Keras, there are two main ways to define models: the Sequential API and the Functional API. In this post, we’ll explore the differences between these two approaches using a simple mathematical example.
Sequential API
The Sequential API in Keras is a linear stack of layers. It’s easy to use but limited to single-input, single-output stacks of layers. Here’s a simple example to illustrate how it works.
Objective:
- Multiply the input \( x \) by 2.
- Add 3 to the result.
Mathematical Operations:
\[ y_1 = 2 \times x \]
\[ y_2 = y_1 + 3 \]
Let’s implement this using the Sequential API:
<br /> from keras.models import Sequential<br /> from keras.layers import Lambda</p> <p># Define a simple sequential model<br /> model = Sequential()<br /> model.add(Lambda(lambda x: 2 * x, input_shape=(1,)))<br /> model.add(Lambda(lambda x: x + 3))</p> <p>model.summary()<br />
Example Calculation:
- If \( x = 1 \):
- \( y_1 = 2 \times 1 = 2 \)
- \( y_2 = 2 + 3 = 5 \)
Functional API
The Functional API in Keras is more flexible and allows for the creation of complex models with multiple inputs and outputs. We’ll use the same mathematical operations to illustrate how it works.
Objective:
- Multiply the input \( x \) by 2.
- Add 3 to the result.
Mathematical Operations:
\[ y_1 = 2 \times x \]
\[ y_2 = y_1 + 3 \]
Let’s implement this using the Functional API:
<br /> from keras.models import Model<br /> from keras.layers import Input, Lambda</p> <p># Define the input<br /> input_ = Input(shape=(1,))</p> <p># Define the operations<br /> y1 = Lambda(lambda x: 2 * x)(input_)<br /> y2 = Lambda(lambda x: x + 3)(y1)</p> <p># Create the model<br /> model = Model(inputs=input_, outputs=y2)</p> <p>model.summary()<br />
Example Calculation:
- If \( x = 1 \):
- \( y_1 = 2 \times 1 = 2 \)
- \( y_2 = 2 + 3 = 5 \)
Summary
Both the Sequential and Functional APIs can achieve the same result in this simple example. However, the Sequential API is limited to linear stacks of layers, while the Functional API allows for more complex architectures. In the next part of this blog post, we’ll dive into a real neural network scenario to further illustrate the differences.
Part 2: Comparing Sequential vs. Functional API with a Real Neural Network Example
In this part, we’ll explore a more complex example involving multiple inputs and outputs to highlight the differences between the Sequential and Functional APIs in Keras.
Real Scenario: Neural Network with Multiple Inputs
Suppose we have two features, \( x_1 \) and \( x_2 \), which represent two different inputs. We want to create a neural network that processes these inputs separately and then combines their outputs to make a final prediction.
Sequential API
The Sequential API does not natively support multiple inputs and outputs. To illustrate, we’ll simplify the scenario by concatenating the inputs first and then processing them.
Sequential Model:
- Concatenate \( x_1 \) and \( x_2 \).
- Process the concatenated input through a Dense layer.
- Produce the final output through another Dense layer.
Implementation:
<br /> from keras.models import Sequential<br /> from keras.layers import Dense, InputLayer</p> <p># Simulate combined input<br /> model = Sequential()<br /> model.add(InputLayer(input_shape=(2,)))<br /> model.add(Dense(units=4, activation='relu'))<br /> model.add(Dense(units=1, activation='sigmoid'))</p> <p>model.compile(optimizer='adam', loss='binary_crossentropy')<br /> model.summary()<br />
Here, we concatenate \( x_1 \) and \( x_2 \) manually before feeding them into the model.
Functional API
The Functional API allows us to directly work with multiple inputs and process them separately before combining the results.
Functional Model:
- Process \( x_1 \) through a Dense layer.
- Process \( x_2 \) through another Dense layer.
- Combine the processed outputs.
- Produce the final output through a Dense layer.
Implementation:
<br /> from keras.models import Model<br /> from keras.layers import Input, Dense, concatenate</p> <p># Define inputs<br /> input_1 = Input(shape=(1,))<br /> input_2 = Input(shape=(1,))</p> <p># Process each input separately<br /> y1 = Dense(units=4, activation='relu')(input_1)<br /> y2 = Dense(units=4, activation='relu')(input_2)</p> <p># Combine the processed inputs<br /> combined = concatenate([y1, y2])</p> <p># Final output layer<br /> output = Dense(units=1, activation='sigmoid')(combined)</p> <p># Create the model<br /> model = Model(inputs=[input_1, input_2], outputs=output)</p> <p>model.compile(optimizer='adam', loss='binary_crossentropy')<br /> model.summary()<br />
Summary of Operations
Sequential API:
- Input: \([x_1, x_2]\)
- Dense layer: \( y = \text{ReLU}(W_1 \cdot [x_1, x_2] + b_1) \)
- Output layer: \( z = \text{sigmoid}(W_2 \cdot y + b_2) \)
Functional API:
- Input 1: \( x_1 \)
- Dense layer: \( y_1 = \text{ReLU}(W_1 \cdot x_1 + b_1) \)
- Input 2: \( x_2 \)
- Dense layer: \( y_2 = \text{ReLU}(W_2 \cdot x_2 + b_2) \)
- Combine: \( y = \text{concatenate}([y_1, y_2]) \)
- Output layer: \( z = \text{sigmoid}(W_3 \cdot y + b_3) \)
Example Calculation
Assume \( x_1 = 1 \), \( x_2 = 2 \), and weights and biases are initialized as follows for simplicity:
- \( W_1 = W_2 = [1, 1, 1, 1] \)
- \( b_1 = b_2 = [0, 0, 0, 0] \)
- \( W_3 = [1, 1, 1, 1, 1, 1, 1, 1] \)
- \( b_3 = 0 \)
Sequential API:
- \( y = \text{ReLU}([1, 1, 1, 1] \cdot [1, 2] + [0, 0, 0, 0]) = \text{ReLU}([3, 3, 3, 3]) = [3, 3, 3, 3] \)
- \( z = \text{sigmoid}([1, 1, 1, 1] \cdot [3, 3, 3, 3] + 0) = \text{sigmoid}(12) \approx 1 \)
Functional API:
- \( y_1 = \text{ReLU}([1, 1, 1, 1] \cdot 1 + [0, 0, 0, 0]) = [1, 1, 1, 1] \)
- \( y_2 = \text{ReLU}([1, 1, 1, 1] \cdot 2 + [0, 0, 0, 0]) = [2, 2, 2, 2] \)
- Combine: \( y = \text{concatenate}([1, 1, 1, 1, 2, 2, 2, 2]) = [1, 1, 1, 1, 2, 2, 2, 2] \)
- \( z = \text{sigmoid}([1, 1, 1, 1, 1, 1, 1, 1] \cdot [1, 1, 1, 1, 2, 2, 2, 2] + 0) = \text{sigmoid}(12) \approx 1 \)
Conclusion
The Functional API provides greater flexibility for handling complex architectures involving multiple inputs and outputs, as demonstrated in this example. It allows you to define models where inputs are processed through separate paths and combined in various ways, which is not directly possible with the Sequential API.
Let’s even compare them in an image
• Diagram: Shows a simple linear stack of layers. • Input Layer: Receives input features (e.g., x1, x2, x3).
• 1st Hidden Layer: Processes the input layer.
• 2nd Hidden Layer: Processes the output from the 1st hidden layer.
• Output Layer: Produces the final prediction (y). • Explanation:
• Sequential API: Simple linear stack of layers, suitable for straightforward models.
Functional API:
• Diagram: Demonstrates a more complex model with branching.
• Input Layer: Receives input features (e.g., x1, x2, x3). • 1st Hidden Layer: Processes the input layer.
• 2nd Hidden Layer: Processes the output from the 1st hidden layer.
• Branched Hidden Layer: Creates a branch for more complex processing.
• 1st Output Layer: Produces the first output (y1).
• 2nd Output Layer: Produces the second output (y2). • Explanation:
• Functional API: Allows complex architectures with multiple inputs, outputs, and shared layers.
This diagram and the accompanying explanations should help illustrate the specific features and use cases of each API more effectively.