bias helps to shift the decision threshold so that the neuron can classify emails correctly even when all the feature values are low
It is a pleasure to guide you through the foundations of neurocomputing. At IISc, we emphasize not just the how, but the why.
To understand the bias term (b), we must first look at the mathematical definition of a Perceptron’s output. A Perceptron takes a set of inputs (x_1, x_2, \dots, x_n), multiplies them by weights (w_1, w_2, \dots, w_n), and sums them up.
The decision rule is generally:
Output = \begin{cases} 1 & \text{if } \sum_{i=1}^{n} w_ix_i + b \geq 0 \\ 0 & \text{if } \sum_{i=1}^{n} w_ix_i + b < 0 \end{cases}
The Problem: The Origin Constraint
Imagine you are building an email classifier. Your features (x) might be “Number of suspicious links” and “Number of urgent keywords.”
If you do not have a bias term (b=0), your decision boundary equation becomes \sum w_ix_i = 0. In a 2D plane, this is a line that must pass through the origin (0,0).
Without a bias, your model is mathematically “locked.” It can only rotate the decision line around the center point. This is a massive limitation:
- What if a “Spam” email has 0 suspicious links and 0 urgent keywords, but is still spam for other reasons?
- What if a “Ham” (good) email has 0 of those features, but your weights are configured such that even a tiny input triggers a “1”?
The Solution: Flexibility via Shifting
The bias allows the decision boundary to move away from the origin.
Think of the weighted sum (\sum w_ix_i) as the “evidence” provided by the data. Think of the bias (b) as the “threshold of skepticism” of the neuron.
- Low Feature Values: Suppose an email arrives with all features near zero. The weighted sum \sum w_ix_i will also be near zero.
- The Role of Bias: * If b is a large positive number, the neuron is “trigger-happy.” Even with zero evidence from features, the result might be > 0, classifying it as 1.
- If b is a large negative number, the neuron is “stubborn.” You need a lot of positive evidence from the features just to overcome the negative bias and reach the threshold of 0.
A Practical Example
Let’s say your “Spam” features are all 0. If your neuron needs to classify this as Spam (1), but your equation is fixed at the origin, you can’t do it because 0+0 will never be greater than 0.
By adding a bias of, say, +5, your equation becomes:
0 + 5 \geq 0
The neuron now outputs 1 (Spam), even though the feature inputs were non-existent.
Key Takeaway
In geometry, y = mx is a line through the origin. y = mx + c is a line that can exist anywhere. In a neural network, bias is your ‘c’. It provides the “flexibility” for the model to represent patterns that don’t naturally align with the zero-point of your feature space.
Certainly. Let us step back into the “Perceptron” mindset—where we use a Step function rather than a Sigmoid. In the IISc curriculum, we define the weight update as a process of “rewarding” the neuron for being correct or “correcting” it when it fails.
The update rule for the bias (b) is remarkably simple because the bias can be thought of as a weight connected to an input that is always 1.
1. The Perceptron Update Formula
When the model makes a mistake, we update the bias using this rule:
b_{new} = b_{old} + \Delta b
The change in bias (\Delta b) is calculated as:
\Delta b = \eta \cdot (Y_{target} – Y_{pred})
Where:
- \eta (Eta) is the Learning Rate (e.g., 0.1).
- Y_{target} is what we wanted (0 or 1).
- Y_{pred} is what the neuron actually output.
2. A Concrete Example
Let’s use the values we discussed earlier to see this in action.
- Current Bias (b_{old}): 0.2
- Learning Rate (\eta): 0.1
- True Label (Y_{target}): 1 (It’s a Dog)
- Model Prediction (Y_{pred}): 0 (The model mistakenly said “Not a Dog”)
Step 1: Calculate the Error
Error = Y_{target} – Y_{pred} = 1 – 0 = \mathbf{1}
Step 2: Calculate the Change (\Delta b)
\Delta b = 0.1 \times 1 = \mathbf{0.1}
Step 3: Apply the Update
b_{new} = 0.2 + 0.1 = \mathbf{0.3}
3. Why did the bias increase?
This is the “intuition” part that distinguishes an elite engineer from a technician.
Remember that the Perceptron fires if z \geq 0. Our model predicted 0, which means the net input (z) was too low (negative).
By increasing the bias from 0.2 to 0.3, we are essentially “lowering the bar.” We are making it easier for the neuron to reach the threshold of 0 next time this specific input appears.
4. What if the model was correct?
If Y_{target} = 1 and Y_{pred} = 1:
- Error = 1 – 1 = 0
- \Delta b = 0.1 \times 0 = 0
- Result: The bias stays exactly where it is. We don’t fix what isn’t broken.
Summary Table: The Bias Update Logic
| Situation | Error (Y−Y^) | Impact on Bias | Why? |
| False Negative | +1 | Increases | Make the neuron more likely to fire. |
| False Positive | -1 | Decreases | Make the neuron “stricter” and less likely to fire. |
| Correct | 0 | No Change | The current threshold is working. |
Now that you see how the bias moves, would you like to see how we update the weights (w_1, w_2) simultaneously? It uses the exact same logic, but with one extra variable!
<Give link to weights blog post here>