[skyrl-train] Flip grpo_norm_by_std default to false

[taivu1998]

Summary

This PR updates the default value of trainer.algorithm.grpo_norm_by_std from true to false across the canonical config and utility surfaces.

It keeps the existing trainer plumbing intact, aligns the config docs with the new default, and adds regression coverage so the repo now explicitly protects:

• the typed config default
• the legacy YAML translation path
• the GRPO utility default behavior
• the trainer forwarding behavior for both default and explicit override cases

Closes #869.

What changed

• Flip AlgorithmConfig.grpo_norm_by_std to False
• Flip the legacy/base YAML default to false
• Flip the GRPO utility defaults in ppo_utils.py to False
• Update the configuration docs and migration wording
• Remove stale hardcoded grpo_norm_by_std=True from shared test helpers
• Add targeted tests for:
  • default behavior equals explicit false
  • explicit true
  • explicit false
  • trainer pass-through behavior

Why this approach

The runtime already forwarded self.cfg.trainer.algorithm.grpo_norm_by_std correctly, so this change stays intentionally small and focused on the default surfaces rather than rewriting trainer logic.

It also avoids broad example churn: examples that already pin false remain explicit, while other flows inherit the new default naturally.

Testing

UV_CACHE_DIR=/tmp/uv-cache uv run --isolated --extra skyrl-train --extra dev pytest -q \
  tests/train/test_config.py \
  tests/train/test_trainer.py \
  tests/backends/skyrl_train/utils/test_ppo_utils.py

UV_CACHE_DIR=/tmp/uv-cache uv run --isolated --extra skyrl-train --extra dev pytest -q \
  tests/train/test_eval.py \
  tests/train/test_trainer_utils.py

---

[gemini-code-assist]

Code Review

This pull request updates the default value of grpo_norm_by_std from true to false across the codebase, including the AlgorithmConfig class, PPO utility functions, and the base YAML configuration. The documentation has been updated to reflect this change and provide instructions for recovering the previous behavior. Additionally, new unit tests have been added to verify the new default setting and ensure that explicit overrides are respected. I have no feedback to provide.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 3 additional findings.