Gradient Clipping in Neural Networks: A Practical Guide to Stable Training

Training a neural network is an iterative process. During every iteration, the model calculates a loss, backpropagation produces gradients, and an optimizer uses those gradients to update the model parameters. Normally, these updates gradually move the model toward a lower loss. Sometimes, however, the gradients become extremely large. The optimizer then makes an excessively large parameter update, which can cause the loss to fluctuate, increase rapidly, or become Inf or NaN. Gradient clipping is a simple technique for limiting these unusually large gradients before the optimizer updates the model. This article explains:

• why large gradients destabilize training

• how gradient norm clipping works

• how it differs from gradient value clipping

• where clipping belongs in the training process

• how to implement it correctly in PyTorch

• how to choose a clipping threshold

• what gradient clipping can and cannot solve

Why can a large gradient become a problem?

Consider the basic gradient-descent update:

θ_new = θ_old − η · ∇θL

where:

• θ represents the model parameters

• η is the learning rate

• L is the loss

• ∇θL is the gradient of the loss with respect to the parameters

Suppose:

θ_old = 2, η = 0.1

If the gradient is 2, the update is:

θ_new = 2 − (0.1)(2) = 1.8

The parameter changes by only 0.2.

Now suppose the gradient suddenly becomes 100:

θ_new = 2 − (0.1)(100) = −8

With the same learning rate, the parameter jumps from 2 to −8. Instead of taking a controlled step, the optimizer moves to a very different region of the parameter space.

Repeated large updates can make optimization unstable. This phenomenon is commonly associated with exploding gradients.

What are exploding gradients?

Exploding gradients occur when gradient magnitudes grow to extremely large values during backpropagation. They are especially well known in recurrent neural networks because gradients may pass through many time steps, but they can also occur in other deep architectures.

Possible warning signs include:

• rapidly increasing gradient norms

• severe fluctuations in the training loss

• sudden divergence after apparently normal training

• model parameters becoming extremely large

• loss or gradients becoming Inf or NaN

Gradient clipping does not remove the underlying reason that gradients became large. Instead, it limits their immediate effect on the parameter update.

What is gradient clipping?

The training sequence without clipping is:

1. Perform a forward pass.

2. Calculate the loss.

3. Run backpropagation.

4. Update the parameters.

With gradient clipping, one additional operation is inserted:

1. Perform a forward pass.

2. Calculate the loss.

3. Run backpropagation.

4. Clip the gradients.

5. Update the parameters.

Therefore, the essential order is:

Backward pass → gradient clipping → optimizer step

Clipping must happen after backpropagation because the gradients do not exist until the backward pass has calculated them. It must happen before the optimizer step so that the optimizer uses the clipped gradients.

Gradient norm clipping

The most widely used form is gradient norm clipping.

Suppose all model gradients are treated as one combined vector:

g = [g₁, g₂, …, gₙ]

Its L₂ norm is:

‖g‖₂ = √(g₁² + g₂² + ⋯ + gₙ²)

Let c be the maximum allowed gradient norm. The clipping operation is:

g_clipped = g, if ‖g‖₂ ≤ c

g_clipped = g · (c / ‖g‖₂), if ‖g‖₂ > c

The gradient remains unchanged when its norm is within the permitted limit. When the norm exceeds the limit, every component is multiplied by the same scaling factor.

An equivalent compact expression is:

g_clipped = g · min(1, c/‖g‖₂)

Because all components receive the same factor, norm clipping reduces the magnitude of the gradient while preserving its direction.

Numerical example

Consider the gradient vector:

g = [6, 8]

Its L₂ norm is:

‖g‖₂ = √(6² + 8²) = √(36+64) = 10

Suppose the maximum permitted norm is:

c = 5

Because 10 > 5, the vector must be scaled:

g_clipped = [6,8] × (5/10)

Therefore:

g_clipped = [3,4]

We can verify the new norm:

√(3² + 4²) = 5

The original vector [6,8] and the clipped vector [3,4] point in the same direction. The clipped vector is simply shorter.

Gradient value clipping

Another method is gradient value clipping. Instead of controlling the norm of the complete vector, this method restricts every individual component to a fixed interval:

−c ≤ gᵢ ≤ c

Equivalently:

gᵢ_clipped = min(max(gᵢ, −c), c)

Using the same gradient:

g = [6,8]

and clipping every value to the interval [−5,5] gives:

g_clipped = [5,5]

Norm clipping versus value clipping

Property	Norm clipping	Value clipping
What is limited?	Norm of the complete gradient vector	Each gradient component
Example for [6,8] with limit 5	[3,4]	[5,5]
Preserves the gradient direction?	Yes	Not necessarily
Typical use	General control of gradient magnitude	Component-wise restrictions

Norm clipping is often a natural default because it controls the overall gradient magnitude without changing the update direction. Value clipping may be appropriate when individual gradient elements must remain within specific bounds.

Implementing gradient norm clipping in PyTorch

PyTorch provides torch.nn.utils.clip_grad_norm_() for norm clipping:

optimizer.zero_grad(set_to_none=True)

outputs = model(inputs)
loss = criterion(outputs, targets)

loss.backward()

torch.nn.utils.clip_grad_norm_(
    model.parameters(),
    max_norm=1.0
)

optimizer.step()

The important sequence is:

loss.backward() clip_grad_norm_(...) optimizer.step()

The trailing underscore in clip_grad_norm_ follows the PyTorch convention for an operation that modifies values in place. In this case, it modifies the gradients stored in the model parameters.

PyTorch computes the total norm as if the individual parameter gradients were combined into one vector. It then scales them when the total norm exceeds max_norm.

Implementing gradient value clipping in PyTorch

For value clipping, PyTorch provides clip_grad_value_():

loss.backward() torch.nn.utils.clip_grad_value_( model.parameters(), clip_value=1.0 ) optimizer.step()

With clip_value=1.0, every gradient component is restricted to the interval [-1,1].

Gradient clipping with mixed-precision training

When using automatic mixed precision with gradient scaling, the gradients should be unscaled before they are clipped:

with torch.autocast(device_type="cuda", dtype=torch.float16): outputs = model(inputs) loss = criterion(outputs, targets) scaler.scale(loss).backward() scaler.unscale_(optimizer) torch.nn.utils.clip_grad_norm_( model.parameters(), max_norm=1.0 ) scaler.step(optimizer)

scaler.update()

Clipping scaled gradients directly would apply the threshold to artificially enlarged values rather than to the actual gradients.

How should the clipping threshold be selected?

The clipping threshold is a hyperparameter. There is no universally correct value.

If the threshold is very large, clipping will rarely activate. If it is too small, many ordinary gradients will be reduced, which may slow learning or interfere with optimization.

A practical procedure is:

1. Monitor the gradient norm during training.

2. Examine its typical values and occasional spikes.

3. Choose an initial threshold.

4. Compare different thresholds using validation performance and training stability.

5. Report the selected value as a training hyperparameter.

A value such as 1.0 is a common experimental starting point, but it is not a general rule.

It is also useful to monitor how frequently clipping occurs. If gradients are clipped during almost every update, the threshold may be too restrictive, or the training process may have another problem that needs investigation.

When is gradient clipping useful?

Gradient clipping can be useful when:

• training recurrent neural networks over long sequences

• training very deep networks

• the loss occasionally produces large gradient spikes

• training becomes unstable even after reasonable learning-rate tuning

• using architectures or objectives known to produce high gradient variance

• working with mixed-precision or large-scale training where numerical stability requires careful monitoring

However, clipping should be introduced because there is a clear stability reason, not simply added to every model without examination.

What gradient clipping can and cannot do

Gradient clipping can:

• limit unusually large parameter updates

• reduce the immediate effect of exploding gradients

• improve training stability

• protect an otherwise useful training run from occasional gradient spikes

Gradient clipping cannot:

• guarantee better validation or test accuracy

• solve vanishing gradients

• correct an unsuitable loss function

• repair incorrect data or labels

• replace appropriate weight initialization

• replace learning-rate tuning

• explain every occurrence of Inf or NaN

If training produces non-finite values, investigate the complete pipeline. Possible causes include an excessive learning rate, unstable mathematical operations, incorrect inputs, an inappropriate loss implementation, mixed-precision issues, or invalid data.

Common mistakes

1. Clipping before backpropagation

2. Before loss.backward(), the current gradients have not been calculated. Clipping at this point cannot control the gradients for the current update.