Gradient Autopsy

The central finding

Depth is the exponent, not the cause

Every experiment here is organized around one number: the per-layer propagation factor, the amount by which a single layer multiplies the backward signal.

A network of depth $L$ multiplies the gradient by roughly ${\chi }^L$ on its way to the input. That single exponent explains the whole corpus. A stack with $\chi =0.98$ is fine at depth 10 and fine at depth 100. A stack with $\chi =0.0448$ is already dead at depth 5.

$\chi$ for every activation and initialization studied

Computed by Gaussian quadrature, not a small-signal approximation. He initialization is exactly the choice that puts ReLU at $\chi =1$.

Bars run from $\chi =1$. Faded bars are within 12% of 1 — the survivable regime. Note that sigmoid cannot be fixed by initialization: $\mathbb{E}[{\sigma }^{\prime }(z{)}^2]\le 1/16$ caps $\chi$ for any input distribution.

Experiment A · 48 runs

Where each activation breaks

Width, optimizer, learning rate and seed held fixed; only depth and the nonlinearity move. Each activation gets the initialization the literature recommends for it.

Sigmoid fails early — and not for the textbook reason

The standard explanation is saturation: units pinned near 0 or 1 where the derivative vanishes. That is not what the instrumentation shows. In the depth-25 sigmoid configuration the measured saturation fraction is 4.9e-5 — essentially no saturated units — while the gradient has already collapsed by 14.8 decades.

The cause is intrinsic and unavoidable: ${\max }_z{\sigma }^{\prime }(z)=1/4$, so $\mathbb{E}[{\sigma }^{\prime }(z{)}^2]\le 1/16$ no matter where the units sit. The contraction is multiplicative, and depth compounds it. Nothing about the input distribution can rescue it.

Gradient propagation ratio, $\Vert {\nabla }_{W_1}\Vert /\Vert {\nabla }_{W_L}\Vert$. Points are missing where the input-side gradient underflowed to exactly zero and the ratio is undefined.

Experiment B · 20 runs

A factor of two, compounded 100 times

Xavier and He differ by exactly $\sqrt{2}$ in weight scale. For ReLU that is $\chi =0.500$ instead of $\chi =1.000$. At depth 10 the difference is invisible. At depth 25 it decides whether the network trains at all.

The two curves are indistinguishable until depth 10, then separate. This is what $0.5^L$ versus $1^L$ looks like in practice.

GELU: He does not transfer

He initialization is derived for ReLU, which is positively homogeneous. GELU is not, so $\mathbb{E}[f(z{)}^2]\ne \frac{1}{2}\mathrm{Var}(z)$ and the variance-preservation argument does not carry over. In this setup $\chi$ drifts from 0.979 at the first layer toward 0.5 deeper in the stack, and the forward variance collapses by layer 100. This is a statement about the configuration measured here, not about every possible GELU setup.

Experiment C · 12 runs

Normalization rescue

Four architectures that Experiments A and B showed to be broken, re-run with BatchNorm and LayerNorm inserted between every linear layer and its activation. On these four selected cases, BatchNorm rescued 3 of 4 and LayerNorm rescued 1.

Experiments D & E · 52 runs

Why the learning rate cannot fix it

A vanishing gradient is often treated as something a bigger step size can compensate for. It cannot. Sweeping $\eta$ across six decades moves the sigmoid stack's propagation ratio essentially not at all — the geometry of the backward pass does not depend on the step size.

Six decades of learning rate; the vanishing stack's ratio stays pinned near -15.03. Raising $\eta$ only buys divergence: the ReLU network produces non-finite losses at $\eta \ge 1$.

Optimizers, each tuned on its own grid

Comparing optimizers at one shared learning rate only measures which optimizer likes that number, so each gets its own grid and is represented by its best result.

Training-free propagation grid

Theory against measurement

No training at all: standardized noise pushed through freshly initialized random networks up to depth 100, measured layer by layer, and compared with the Gaussian variance recursion.

Both expectations are evaluated by quadrature, so the prediction is exact under the Gaussian assumption rather than a small-signal linearization. It tracks the measurement across the full dynamic range the corpus spans — tens of decades of decay, and tens of decades of growth.

The residual is a finite-width effect

The recursion is a large-width limit; at finite width a single network's log-variance performs a random walk around it. If the framework is right, the residual should shrink as the network widens — and it does.

tanh/Xavier falls from 0.198 at $n=32$ to 0.006 at $n=1024$. ReLU/He is the marginal $\chi =1$ case, where the log-variance walk has no restoring drift, so its residual shrinks more slowly and less smoothly.

Diagnosis engine

Automatic autopsy

Every run is diagnosed by a deterministic, rule-based engine — no model, no LLM. Thresholds are declared constants, verdicts are auditable, and the whole corpus can be re-diagnosed from disk without retraining. Across 132 runs it returned 56 healthy, 16 degraded and 60 failed.

AUTOPSY RESULT
--------------
Run: A_depth_activation__matched_sigmoid_d25__5e0b80effe
Status: Failed

Architecture:
25 x 256 sigmoid, init=xavier_normal, norm=none, sgd lr=0.05

Gradient attenuation:
output -> input gradient ratio = 1.58e-15 (log10 = -14.80)
||grad W_first|| = 1.43e-15, ||grad W_last|| = 7.92e-01

Activation propagation:
Var(h_L)/Var(h_1) = 2.89e-01 (log10 = -0.54)

Parameter updates:
first 18 layers update/weight < 1e-07
Parameter updates have stalled in 18/26 linear layers (update/weight < 1e-07); the first 18 layers are frozen.

Learning outcome:
validation accuracy 0.0930 (chance 0.1000), final train loss 2.3173

Findings:
  [critical] vanishing_gradient: Gradient attenuates toward the input: output -> input gradient ratio = 1.58e-15.
  [critical] stalled_updates: Parameter updates have stalled in 18/26 linear layers (update/weight < 1e-07); the first 18 layers are frozen.

Three rules that came from the data

These exist because real experiment output broke simpler assumptions — the part of the work that only shows up once you run the corpus.

• Gradient extinction. When the input-side gradient underflows to exactly zero in float32 while the output side is still alive, ${\log }_{10}$ of the ratio is undefined and a plain ratio test silently misses the most severe case there is. Extinction is now detected and reported separately from ordinary vanishing. 7 runs in the corpus are in this state.
• Trigger disambiguation. A finding raised by an absolute gradient norm must not claim that a ratio exploded. One Adam run reported “amplifies toward the input” alongside a ratio of 0.25; the rule now names which test actually fired.
• Partial learning. A network that crawls above chance is not healthy. A run whose validation loss falls to only a modest fraction of its starting value is reported as partial_learning rather than waved through — 10 runs land here.

Boundaries

What this does not show

• Architecture. MLPs only, on flattened images, with no skip connections — the standard fix for exactly this problem, deliberately out of scope. These findings do not automatically transfer to residual networks, convolutional networks or Transformers.
• Data. Fashion-MNIST and a synthetic Gaussian mixture, not large modern benchmarks.
• The $\chi$ framework rests on mean-field variance-propagation assumptions — independent weights, Gaussian pre-activations by CLT over the width. It describes this regime well; it is not offered as an architecture-independent law.
• Finite width. The recursion is a large-width limit; deviations at width 256 are quantified in the width sweep rather than hidden.
• Normalization conclusions come from four selected failure cases, not a general comparison of BatchNorm and LayerNorm.
• Optimizer results depend on implementation and hyperparameters — PyTorch defaults, per-optimizer learning-rate grids, a five-epoch budget.
• Numerics. float32 throughout, with subnormals deliberately preserved rather than flushed, so recorded gradients are the true float32 values. The associated runtime behaviour is CPU-dependent.
• Budget. One seed per trained configuration. The conclusions concern where propagation breaks, not final attainable accuracy.

Reproduce

Verification in three minutes, or the full corpus in 4.3 hours

The fast path needs no training corpus: it runs the test suite, exercises the entire pipeline on four tiny networks, and re-checks every number quoted in the report against the generated tables.

git clone github.com/Gariyuuu/gradient-autopsy
cd gradient-autopsy && pip install -r requirements.txt && pip install -e .

pytest -q                            # 119 tests
python scripts/run_smoke.py          # end-to-end pipeline check, ~5 s
python scripts/verify_artifacts.py   # corpus + documented-number consistency

The full corpus, if you want it — about 4.3 hours of wall clock on the documented CPU, of which 4.2 h is recorded training time and the rest is the training-free theory grid and initialization audit:

python -m gradient_autopsy.experiments.run_all   # 132 runs
python -m gradient_autopsy.figures               # rebuild all figures
streamlit run app/streamlit_app.py               # interactive explorer