PerNodeDrop: A Practical and Efficient Alternative to DropConnect:- – Exploring Node-Owned Stochasticity

[geleshChrsitUniversity]

PerNodeDrop — a per-sample, per-node stochastic regularizer (Keras 3 layer)

Idea # 23090

Hi everyone! Sharing a small regularization technique I've been working on, along with a Keras-native layer implementation.

The idea

Standard noise-based regularizers like Dropout and DropConnect inject stochastic perturbations during training, but the noise is typically uniform across a layer or across a batch of samples. That uniformity suppresses both harmful and beneficial co-adaptation between neurons — which can leave useful, fine-grained feature interactions on the table.

PerNodeDrop applies perturbations at a finer grain: per-sample, per-node, rather than per-batch or per-layer. Each node experiences input-specific variability, so the model can retain useful co-adaptation while still getting the overfitting-reducing benefits of stochastic regularization.

It's worth being precise about where this sits relative to DropConnect, since there are two versions worth distinguishing:

• DropConnect, original (Wan et al., 2013) — technically is per-sample, applying a distinct mask to individual weights (connections) for every training example. In principle this is finer-grained than PerNodeDrop. In practice, though, it's computationally demanding: sampling a unique mask per example means materializing a distinct, high-dimensional weight matrix per sample, requiring custom grid/kernel handling that doesn't scale cleanly on standard hardware. The original paper even resorts to a Gaussian moment-matching approximation just to make inference tractable.
• DropConnect, common implementations (e.g. typical PyTorch ports) — sidestep that cost by sampling one mask per batch rather than per example, trading away the original method's per-sample granularity for tractability.

PerNodeDrop aims at a middle ground: it keeps the per-sample property that the original DropConnect has (which batch-level approximations lose), but perturbs at the coarser node level rather than every individual connection — avoiding the memory blow-up and high-dimensional matrix handling that makes the exact Wan et al. formulation expensive.

In short:

Method	Granularity	What's perturbed	Computational cost
Dropout	per-batch / per-layer	units	Low
DropConnect (Wan et al., 2013)	per-sample, per-connection	individual weights	High — custom per-example weight matrices, high-dimensional
DropConnect (typical PyTorch impl.)	per-batch	individual weights	Low — but loses per-sample granularity
PerNodeDrop	per-sample, per-node	weights (grouped by node)	Moderate — per-sample without full per-connection cost

Usage

It's a drop-in Keras layer:

import keras
from pernodedrop import PerNodeDrop  # pip install / see repo

model = keras.Sequential([
    keras.layers.Dense(256, activation="relu"),
    
    #Dropout(rate=0.3),
    #keras.layers.Dense(128, activation="relu"), 
    
    # This combines a Dropout and Dese , but the mask applied to each of 128 is independent 
    PerNeuronDropDense(128, drop_rate=0.3, stir_type="drop", activation="relu"),

    keras.layers.Dense(10, activation="softmax"),
])

model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])

Works with the standard Keras 3 backends (TensorFlow / JAX / PyTorch) — [ADD: confirm which backends you've tested].

Results

Evaluated on vision, text, and audio benchmarks against standard Dropout / DropConnect baselines:

Dataset / Task	Baseline (Dropout)	PerNodeDrop	Train/Val gap (baseline → ours)
[ADD: e.g. CIFAR-10, ResNet-18]	[ADD acc]	[ADD acc]	[ADD gap]
[ADD: e.g. IMDB sentiment]	[ADD acc]	[ADD acc]	[ADD gap]
[ADD: e.g. Speech Commands]	[ADD acc]	[ADD acc]	[ADD gap]

Across these, PerNodeDrop narrows the gap between training and validation performance compared to standard noise-based regularization, suggesting better reliability on unseen data.

Links

• 🔗 Repo: https://github.com/geleshChrsitUniversity/PerNodeDrop
• 📄 Write-up / paper: [ADD link if available]
• ▶️ Colab demo: [ADD link if available — highly recommend adding one, gets much more engagement]

Feedback welcome

This is early-stage — I'd love feedback on:

• Whether the per-sample/per-node approach makes sense theoretically
• Interest in trying it on attention blocks / transformers
• Any issues getting it running with different Keras backends

Happy to answer questions or take PRs on the repo!