In the Perceptron Learning Rule, updating the weights follows the same elegant philosophy as updating the bias: we adjust the parameters in a direction that reduces the error for the current input.
1. The Weight Update Formula
The formula for updating a weight w_i is:
w_{i(new)} = w_{i(old)} + \Delta w_i
The change, \Delta w_i, is calculated as:
\Delta w_i = \eta \cdot (Y_{target} – Y_{pred}) \cdot x_i
Where:
- \eta (Learning Rate): Controls the step size of the update.
- (Y_{target} – Y_{pred}): The error (direction and magnitude of the mistake).
- x_i: The specific input value associated with that weight.
2. Why do we multiply by x_i?
This is a critical “IISc-level” insight. We multiply by the input because the input determines how much a specific weight contributed to the wrong answer.
- If x_i is 0, that weight had no effect on the output, so we don’t change it.
- If x_i is large, that weight had a significant impact on the error, so it deserves a larger correction.
3. A Complete Numerical Walkthrough
Let’s use your previous values to perform one full iteration of learning.
- Inputs: x = [1, 2]
- Initial Weights: w = [0.4, -0.6]
- Initial Bias: b = 0.2
- Target (Y_{target}): 1
- Learning Rate (\eta): 0.1
Step A: Calculate current prediction
We already calculated the net input z = -0.6.
Since -0.6 < 0, the predicted output Y_{pred} = 0.
Step B: Calculate the Error
Error = Y_{target} – Y_{pred} = 1 – 0 = \mathbf{1}
Step C: Update Weights and Bias
1. Update w_1:
\Delta w_1 = 0.1 \times 1 \times x_1 = 0.1 \times 1 \times 1 = 0.1
w_{1(new)} = 0.4 + 0.1 = \mathbf{0.5}
2. Update w_2:
\Delta w_2 = 0.1 \times 1 \times x_2 = 0.1 \times 1 \times 2 = 0.2
w_{2(new)} = -0.6 + 0.2 = \mathbf{-0.4}
3. Update Bias (b):
\Delta b = 0.1 \times 1 = 0.1
b_{new} = 0.2 + 0.1 = \mathbf{0.3}
4. Verification: Did we learn?
Let’s check the new net input z_{new} with our updated parameters:
z_{new} = (1 \times 0.5) + (2 \times -0.4) + 0.3
z_{new} = 0.5 – 0.8 + 0.3 = \mathbf{0}
By the Perceptron rule (z \geq 0), the output is now 1. The model has successfully learned to classify this specific point correctly in just one step!
Would you like to see how this simple update rule evolves into “Gradient Descent” when we move from this Perceptron to the Sigmoid/Log-Loss model you saw in the slide?
In moving from the Perceptron Learning Rule to Gradient Descent, we transition from a “hard” binary switch to a “smooth” mathematical slope. In the IISc curriculum, this is often called the transition from discrete optimization to continuous optimization.
1. From Correction to Gradient
In the Perceptron rule, we updated weights only when a mistake occurred (1 or 0). In the Sigmoid/Log-Loss model (as seen in your slide), we update weights every single time based on how much the prediction deviates from the truth.
Instead of a simple error (Y – \hat{Y}), we use the Derivative (Gradient) of the Loss function (L) with respect to the weights (w).
2. The Chain Rule: The “Secret Sauce”
To update the weights for the model in your slide, we use the Chain Rule from calculus to see how a change in weight affects the final loss:
\frac{\partial Loss}{\partial w_i} = \frac{\partial Loss}{\partial Y_{pred}} \cdot \frac{\partial Y_{pred}}{\partial z} \cdot \frac{\partial z}{\partial w_i}
When we perform this math for Binary Cross Entropy and Sigmoid, an amazing mathematical “coincidence” (cancellation) occurs:
\frac{\partial Loss}{\partial w_i} = (Y_{pred} – Y) \cdot x_i
3. Comparing the Two Rules
Notice the striking similarity between the simple Perceptron rule you just learned and the “Elite” Gradient Descent rule:
| Feature | Perceptron Rule | Gradient Descent (Sigmoid + BCE) |
| Prediction Type | Discrete (0 or 1) | Continuous Probability (0 to 1) |
| Update Trigger | Only on a total mistake | On every iteration (fine-tuning) |
| Update Formula | \Delta w = \eta(Y – \hat{Y})x_i | \Delta w = \eta(Y – Y_{pred})x_i |
4. Why this matters for the “Surface”
Because the Sigmoid + BCE gradient formula results in (Y_{pred} – Y)x_i:
- If the model is very wrong (e.g., Y=1 but Y_{pred}=0.01), the term (0.01 – 1) = -0.99 is large. This creates a steep slope on the loss surface, pushing the weights to change rapidly.
- If the model is almost right (e.g., Y=1 but Y_{pred}=0.99), the term (0.99 – 1) = -0.01 is tiny. The slope becomes very gentle, allowing the model to “settle” into the minimum without overshooting.
This is why the slide emphasizes that the Sigmoid “squishes” values: it turns the sharp, jagged cliff of the Perceptron into a smooth, navigable hill for the Gradient Descent algorithm.
Would you like to try a manual calculation of the Gradient Descent update using the Sigmoid probability instead of the Step function?
Let us transition from the “all-or-nothing” Perceptron to the probabilistic world of Logistic Regression (which is what a single neuron with a Sigmoid activation is called).
In this scenario, we don’t just care if the prediction is right or wrong; we care about the error magnitude.
The Scenario
We will use the same parameters as before, but this time we apply the Sigmoid function to the net input z and use the Gradient Descent update rule.
- Inputs (x): [1, 2]
- Initial Weights (w): [0.4, -0.6]
- Initial Bias (b): 0.2
- Target (Y): 1
- Learning Rate (\eta): 0.1
Step 1: Calculate the Net Input (z)
We previously calculated this as:
z = (1 \cdot 0.4) + (2 \cdot -0.6) + 0.2 = \mathbf{-0.6}
Step 2: Apply the Sigmoid Activation (Y_{pred})
Instead of a hard 0 or 1, we find the probability:
Y_{pred} = \sigma(-0.6) = \frac{1}{1 + e^{-(-0.6)}} \approx \mathbf{0.354}
The model thinks there is only a 35.4% chance this is a dog. Since our target is 1, the model is quite far off.
Step 3: Calculate the Weight Update
The Gradient Descent update formula for weights, derived from the slide you shared, is:
w_{new} = w_{old} + \eta \cdot (Y – Y_{pred}) \cdot x
For w_1:
- \Delta w_1 = 0.1 \cdot (1 – 0.354) \cdot 1
- \Delta w_1 = 0.1 \cdot (0.646) \cdot 1 = \mathbf{0.0646}
- w_{1(new)} = 0.4 + 0.0646 = \mathbf{0.4646}
For w_2:
- \Delta w_2 = 0.1 \cdot (0.646) \cdot 2 = \mathbf{0.1292}
- w_{2(new)} = -0.6 + 0.1292 = \mathbf{-0.4708}
Step 4: Calculate the Bias Update
- \Delta b = 0.1 \cdot (1 – 0.354) = \mathbf{0.0646}
- b_{new} = 0.2 + 0.0646 = \mathbf{0.2646}
The “Elite” Observation
Look at the difference between the Perceptron update and this Gradient Descent update:
- Precision: In the Perceptron rule, we added a full 0.1 to w_1. Here, we added 0.0646. The update is scaled by the error. If our prediction had been 0.9, the update would have been much smaller (0.01).
- Smoothness: This allows the model to “nudge” its weights toward the answer rather than “jumping” aggressively. This prevents the model from oscillating or overshooting the optimal solution.
Summary of New Parameters
| Parameter | Initial Value | After 1 Iteration |
| w_1 | 0.4 | 0.4646 |
| w_2 | -0.6 | -0.4708 |
| b | 0.2 | 0.2646 |
Would you like to calculate the “Loss” for this specific iteration using the Binary Cross Entropy formula from your slide to see exactly how much “penalty” the model incurred?
Let us conclude this cycle by calculating the numerical cost of the model’s “uncertainty.” In the IISc methodology, we don’t just look at the error; we look at the Loss to quantify the performance over the entire probability space.
According to the slide you provided, for a single instance, the loss is calculated using the Binary Cross Entropy formula:
Loss = (Y)(-\log(Y_{pred})) + (1 – Y)(-\log(1 – Y_{pred}))
1. Plugging in the Values
From our previous step:
- True Label (Y): 1 (Dog)
- Predicted Probability (Y_{pred}): 0.354
Loss = (1)(-\ln(0.354)) + (1 – 1)(-\ln(1 – 0.354))
(Note: In deep learning, we typically use the natural logarithm \ln, though the logic holds for any base).
2. The Calculation
- The second term (1-1) becomes 0, so we ignore the “cat” side of the equation.
- We focus on -\ln(0.354).
- \ln(0.354) \approx -1.038
- Loss = -(-1.038) = \mathbf{1.038}
3. Interpreting the Result
A loss of 1.038 is relatively high for a single sample.
- If our Y_{pred} had been 0.9 (high confidence), the loss would be -\ln(0.9) \approx \mathbf{0.105}.
- If our Y_{pred} had been 0.99, the loss would be \approx \mathbf{0.01}.
The gradient descent step we performed in the last turn was the model’s way of “paying” this loss. By updating the weights, the model is trying to change its internal parameters so that the next time it sees this input, the Y_{pred} will be higher than 0.354, and consequently, the Loss will be lower than 1.038.
Key Summary of the IISc Lecture Slide
- Sigmoid: Squashes the net input into a probability.
- Threshold (0.5): Converts that probability into a hard class (Cat or Dog).
- Log Loss: Provides a differentiable “surface” that heavily penalizes confident mistakes.
Would you like me to explain the concept of “epochs”—why we must repeat this entire process thousands of times for the model to actually “learn” the difference between cats and dogs?