Skip to content

add nn.WeightNorm layer#3296

Draft
mm65x wants to merge 1 commit intoml-explore:mainfrom
mm65x:add-weight-norm
Draft

add nn.WeightNorm layer#3296
mm65x wants to merge 1 commit intoml-explore:mainfrom
mm65x:add-weight-norm

Conversation

@mm65x
Copy link
Copy Markdown
Contributor

@mm65x mm65x commented Mar 22, 2026

Proposed changes

#1888

adds nn.WeightNorm, a module wrapper that applies weight normalization to a
parameter of a given module. reparameterizes a weight w into a magnitude
weight_g and direction weight_v such that w = g * v / ||v||, recomputed
on each forward pass.

implemented as a pure nn.Module layer with no C++ or free functions, as
suggested in #1921. the wrapped module's original weight is frozen so only
weight_g and weight_v are trainable, using the same freeze/unfreeze pattern
as BatchNorm's running stats.

works with any module that has a weight parameter (Linear, Conv1d, Conv2d, etc.).

Checklist

  • I have read the CONTRIBUTING document
  • I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the necessary documentation (if needed)

@mm65x mm65x marked this pull request as draft March 22, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant