Explaining Regularization L1, L2 and ElasticNet

The model that aced the practice paper

Here's an experiment you can run in a minute. Make a dataset with 39 features and only 60 rows. Five features actually drive the target; the other 34 are pure random noise. Fit ordinary linear regression.

On the 60 training rows, the error (mean squared error) is 1.82. On 5,000 fresh rows from the same source, it's 9.51. That's more than five times worse.

The model didn't learn the pattern. It learned the training set: every accidental wiggle in those 34 noise columns got its own little weight, tuned to push the 60 predictions exactly where they needed to go. It's the student who memorises last year's answers and then sits a different paper.

This is overfitting, and it gets worse whenever you have many features, few rows, or features that echo each other. Regularization is the standard fix. This post walks through the three classic flavours, Ridge, Lasso and Elastic Net, using an implementation I wrote from scratch so every step is visible.

A tax on big weights

Plain linear regression picks weights W and a bias b to minimise one thing: how wrong the predictions are on the training data.

Regularization adds a second term, a penalty that grows with the size of the weights:

Now every weight has to earn its keep. A weight can only grow if it reduces the error by more than it costs in tax. Real signal easily pays the tax. The noise columns, which only helped on 60 lucky rows, mostly can't.

λ (lambda) sets the tax rate. At λ = 0 you're back to plain regression. Crank it up and every weight is pushed toward zero; crank it too far and the model can't even learn the real pattern. You're trading a little bias for a lot less variance, and λ is where you choose the trade.

Two rules apply to all three methods:

1. Don't tax the bias. The bias b just shifts every prediction up or down. Shrinking it toward zero would drag your predictions toward zero for no reason. In the code, the bias always gets a plain gradient update.
2. Standardize features first. The penalty treats every weight the same. If one feature is measured in metres and another in millimetres, the millimetre one needs a weight 1,000 times smaller and gets taxed far less. Run StandardScaler before fitting, every time.

The three methods differ only in what penalty(W) is. That one choice changes their behaviour completely.

Ridge (L2): everyone shrinks a little

Ridge taxes the square of each weight:

Squares are smooth, so Ridge has a clean closed-form answer. Set the gradient to zero and solve:

X_b is X with a column of ones stuck on the front, so the first entry of the answer is the bias. I' is the identity matrix with its top-left entry set to zero, which is how the bias escapes the tax.

X_b = np.hstack((np.ones((X.shape[0], 1)), X))
penality = lam * np.eye(X_b.shape[1])
penality[0, 0] = 0          # don't regularize the bias
self._W = np.linalg.inv(X_b.T @ X_b + penality) @ X_b.T @ y
self._b = self._W[0]
self._W = self._W[1:]

There's a bonus hiding in that formula. When two features are perfectly collinear, X_bᵀX_b can't be inverted and plain regression breaks. Adding λI' to the diagonal makes it invertible again. Ridge was originally invented for exactly this problem.

The same class can also train with gradient descent (pass iterations > 0). The L2 penalty adds λW to the weight gradient:

dW = (np.dot(X.T, (y_h - y)) + self._lambda * self._W) / m
db = np.sum(y_h - y) / m
self._W = self._W - self._alpha * dW
self._b = self._b - self._alpha * db

Rearrange that update and you get:

Before each normal gradient step, every weight is multiplied by a number slightly below 1. That's why deep learning people call L2 regularization weight decay: the weights slowly decay unless the data keeps pushing them back up.

Notice what this means for small weights. The shrink is proportional. A weight of 10 loses a bit; a weight of 0.01 loses a tiny bit. Ridge weights get very small, but they essentially never become exactly zero. Ridge keeps every feature.

Lasso (L1): a constant pull that reaches zero

Lasso taxes the absolute value of each weight:

The gradient of |w| is just +1 or -1: a constant-strength pull toward zero, no matter how small the weight is. Compare that to Ridge, where the pull fades as the weight shrinks. Lasso's pull doesn't fade, so a small weight gets dragged all the way to zero and stays there.

That's Lasso's superpower: it deletes features. A weight of exactly zero means that feature isn't used at all. You get a sparse model and free feature selection.

There's a catch. |w| has a sharp corner at zero, where the gradient isn't defined. If you try ordinary gradient descent, a weight near zero gets pushed to +0.001, then to -0.001, then back, bouncing around zero forever without ever landing on it.

My implementation uses proximal gradient descent (also called ISTA) instead. Each step is split in two:

1. Take a normal gradient step on the smooth part (the error) only.
2. Apply the L1 penalty separately with a soft-threshold: move each weight toward zero by α·λ, and if that would carry it past zero, set it to exactly zero.

def _soft_threshold(self, w, thresh):
    return np.sign(w) * np.maximum(np.abs(w) - thresh, 0.0)

# inside the training loop
dW = np.dot(X.T, (y_h - y)) / m
db = np.sum(y_h - y) / m
w_temp = self._W - self._alpha * dW                                # 1. step on the error
self._W = self._soft_threshold(w_temp, self._alpha * self._lambda)  # 2. shrink or zero
self._b = self._b - self._alpha * db                               # bias: plain update

Soft-thresholding is a small function with a big job. Any weight whose size is under the threshold snaps to exactly 0. Everything else slides toward zero by the same fixed amount. No bouncing, and real zeros.

Elastic Net: both taxes, with a dial

Elastic Net charges both taxes at once and lets you set the mix with l1_ratio:

At l1_ratio = 1 it's pure Lasso; at 0 it's pure Ridge. In between, you get Lasso's ability to zero out features plus Ridge's steadiness.

The proximal step handles both parts. Soft-threshold for the L1 part, then divide by a number slightly above 1 for the L2 part:

l1 = self._lambda * self._l1_ratio
l2 = self._lambda * (1 - self._l1_ratio)

w_temp = self._W - self._alpha * dW
self._W = self._soft_threshold(w_temp, self._alpha * l1) / (1 + self._alpha * l2)

That division is weight decay again, written in its proximal form.

Why bother mixing? Because of correlated features. Suppose two columns carry nearly the same information, like "house size in m²" and "number of rooms". Lasso tends to keep one and zero the other, and which one it keeps can flip with a small change in the data. Elastic Net's L2 part makes it prefer sharing the weight, so correlated features tend to stay or go together.

Does it actually work? Checking against scikit-learn

Writing an algorithm from scratch is only useful if it's right, so each file ends by fitting scikit-learn's version on the same data: 3,000 rows, 39 standardized features, seed 45.

That Lasso result looks like a bug at first. Every single weight is zero?

It's actually the best answer. In these demos the target is random numbers, completely unrelated to the features. Lasso looked at all 39, decided none were worth the tax, and fell back to predicting the average (0.5009). That's a model honestly saying "nothing here helps". Plain regression would have happily handed you 39 confident-looking weights built from noise.

Elastic Net with a weaker penalty lands in between: a handful of tiny weights survive and the rest are exactly zero.

The real test: 5 signals hiding in 39 columns

Random-noise targets prove the code matches, but they can't show which method is better. So back to the experiment from the start of the post:

• 39 features, 60 training rows.
• The target is 4·x₀ + 3·x₁ − 2·x₂ + 1.5·x₃ + 1·x₄, plus noise.
• Feature x₅ is a near copy of x₀ (its "twin"), but has no real effect on the target.
• The other 33 features are pure noise.
• For each method, λ was picked on a separate validation set, then everything was scored on 5,000 unseen test rows.

The noise alone gives a test error floor of about 4.0. No model can beat that.

Four things stand out:

1. Training error goes up, test error goes down. Every regularized model fits the training set worse than plain regression and the unseen data better. That's the whole point: you stop rewarding memorisation.
2. Plain regression got confused by the twin. The true weight on x₀ is 4 and on x₅ is 0. Plain regression gave the twin more weight than the real feature (2.68 vs 1.21), because with 60 rows it couldn't tell them apart.
3. Ridge split the difference. 1.83 and 1.85, almost perfectly shared. That's Ridge's signature with correlated features: it spreads the weight evenly instead of picking.
4. Lasso found the real one. It put 3.21 on x₀, squeezed the twin down to 0.24, and dropped 20 features entirely.

Lasso wins here because the truth really is sparse: a few features matter and the rest are junk. That's exactly the world Lasso assumes. Elastic Net keeps the twin around (1.73 and 1.61), which cost it a little here. If both twins were genuinely useful, keeping them together would be the safer behaviour.

Watching Lasso decide

The fun part of Lasso is turning up λ and watching features drop out one by one. Same data, same code, just a bigger tax each time:

At λ = 0.5, all five real features survive, the twin is gone, and only three noise columns are still hanging on. Push further and Lasso starts sacrificing real features too, weakest first: x₃ goes at λ = 1, then x₂ and x₄. By λ = 4 the tax is so high that nothing is worth keeping.

The order features drop out in is itself useful. It's a ranking of how much each feature matters, for free.

Which one should you reach for?

A practical rule of thumb:

• Ridge when you think most features carry some signal, or you have lots of correlated features and just want a stable model.
• Lasso when you suspect only a few features matter and you want to know which ones.
• Elastic Net when you have many features, some correlated, and you still want feature selection without Lasso's coin-flipping between twins. If you're unsure, it's a sensible default.

Gotchas I hit along the way

1. Forgetting to scale. Without standardization, the penalty punishes features for their units, not their usefulness. Every demo in the repo runs StandardScaler first.
2. Taxing the bias. Easy to do by accident in the closed form. Zero out penality[0, 0].
3. Using plain gradient descent for Lasso. Weights shimmer around zero instead of landing on it, and you lose the sparsity that was the whole reason to use Lasso. Use a proximal step.
4. Mixing up alpha. Learning rate in my code; regularization strength in scikit-learn.
5. Picking λ on the training set. Training error always prefers λ = 0. Choose λ on a validation set or with cross-validation, as in the experiment above.
6. Trusting Lasso's pick between twins. If two features are highly correlated, the one Lasso keeps isn't necessarily the "real" one. Check with domain knowledge, or use Elastic Net.

Takeaways

1. Overfitting shows up as a big gap between training and test error. In the experiment, it was 1.82 vs 9.51.
2. Regularization adds a tax on weight size, so only features that genuinely reduce error can afford large weights.
3. Ridge shrinks proportionally and keeps everything. Lasso pulls with a constant force and zeros features out. Elastic Net lets you blend the two.
4. Lasso needs a proximal step (soft-thresholding) to reach exact zeros; ordinary gradient descent just bounces around zero.
5. Always standardize, never tax the bias, and tune λ on data the model hasn't trained on.
6. Building these from scratch and checking against scikit-learn is one of the best ways to really understand them. The full code is in the repo.

Written by Abhi Sharma

Data and AI @ Google | I love to write, explain and build things.