Skip to content

fix pred_std scaling#523

Open
Debadri-das wants to merge 1 commit intomllam:mainfrom
Debadri-das:fix-pred-std-scaling
Open

fix pred_std scaling#523
Debadri-das wants to merge 1 commit intomllam:mainfrom
Debadri-das:fix-pred-std-scaling

Conversation

@Debadri-das
Copy link
Copy Markdown

Describe your changes

Immediately after softplus, scale the predictor standard deviation (pred_std) using the one-step difference statistics (self.diff_std).

Instead of beginning around ~0.69, this first scales pred_std to the empirical step differences. This keeps measurements like NLL and CRPS from experiencing logarithmic loss explosions in the early stages of training. This offers a better initialization scale, as was discussed with @joeloskarsson. We consider this to be the main initialization step; training curve evaluation is necessary to determine whether the suggested / softplus(0.) scaling multiplier will be needed as a follow-up.
None.

Issue Link

linked to #347

Type of change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📖 Documentation (Addition or improvements to documentation)

Checklist before requesting a review

  • My branch is up-to-date with the target branch - if not update your fork with the changes from the target branch (use pull with --rebase option if possible).
  • I have performed a self-review of my code
  • For any new/modified functions/classes I have added docstrings that clearly describe its purpose, expected inputs and returned values
  • I have placed in-line comments to clarify the intent of any hard-to-understand passages of my code
  • I have updated the README to cover introduced code changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have given the PR a name that clearly describes the change, written in imperative form (context).
  • I have requested a reviewer and an assignee (assignee is responsible for merging). This applies only if you have write access to the repo, otherwise feel free to tag a maintainer to add a reviewer and assignee.

Checklist for reviewers

Each PR comes with its own improvements and flaws. The reviewer should check the following:

  • the code is readable
  • the code is well tested
  • the code is documented (including return types and parameters)
  • the code is easy to maintain

Author checklist after completed review

  • I have added a line to the CHANGELOG describing this change, in a section reflecting type of change, please add where missing.

@Debadri-das Debadri-das force-pushed the fix-pred-std-scaling branch from 93311d3 to 99fb1ec Compare March 27, 2026 06:28
Scale pred_std by diff_std to prevent probabilistic loss from exploding when output_std=True.

Closes mllam#347

Co-authored-by: GitHub Copilot <noreply@github.com>
@Debadri-das Debadri-das force-pushed the fix-pred-std-scaling branch from 99fb1ec to f67dd44 Compare March 27, 2026 06:31
@Debadri-das
Copy link
Copy Markdown
Author

Hi @sadamov @joeloskarsson , I noticed this issue regarding pred_std scaling and have created a preliminary PR adjusting the initialization to self.diff_std. Could you please review this?

@Debadri-das Debadri-das changed the title Fix pred std scaling [fix] pred std scaling Mar 27, 2026
@Debadri-das Debadri-das changed the title [fix] pred std scaling fix pred std scaling Mar 27, 2026
@Debadri-das Debadri-das changed the title fix pred std scaling fix pred_std scaling Mar 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant