test: extract enforce_distribution / enforce_sorting tests into a dedicated crate

[zhuqi-lucas]

Which issue does this PR close?

Prep PR for #21976 (EnsureRequirements). Splits the test reorganization @alamb requested in his review (#21976 (review)) so the EnsureRequirements diff stays focused on the rule change.

Rationale

When EnsureRequirements lands and replaces EnforceDistribution + EnforceSorting, we want the diff to show:

1. the existing test files now calling EnsureRequirements::new() instead of the old rules (function-name swap on already-moved files), and
2. any genuinely new tests for new behavior.

Today the four big test files live in core/tests/physical_optimizer/, while #21976 introduces a separate ensure_requirements/new_tests.rs plus inline tests in ensure_requirements/mod.rs. By moving the test files now, the follow-up rebase of #21976 will produce exactly the diff @alamb asked for.

What this PR does

Moves four integration test files into a new workspace crate datafusion-physical-optimizer-tests (publish = false):

File	Tests
enforce_distribution.rs	66
enforce_sorting.rs	60
enforce_sorting_monotonicity.rs	64
replace_with_order_preserving_variants.rs	46

Git detected the four moves as renames with 93–98% content similarity, so reviewers can use the rename view to confirm the test bodies didn't change.

Why a new crate (not datafusion/physical-optimizer/tests/)

The simpler "move to physical-optimizer/tests/" approach was the initial plan, but it would require adding datafusion = { workspace = true } as a dev-dependency of datafusion-physical-optimizer — these tests use datafusion::prelude::SessionContext. Since datafusion (core) already runtime-depends on datafusion-physical-optimizer, this creates a cycle that the project's dev/depcheck rejects (it guards crates.io publish ability).

datafusion-physical-optimizer-tests is a leaf in the dependency graph — nothing depends on it — so its dev-deps on datafusion and datafusion-physical-optimizer introduce no cycle. cargo metadata confirms zero incoming edges. The crate is publish = false, following the precedent of test-utils and datafusion-wasmtest.

Mechanical changes

• Cargo.toml (workspace): adds datafusion/physical-optimizer-tests to members.
• datafusion/physical-optimizer-tests/Cargo.toml: declares dev-dependencies the tests need. Crate has no runtime deps; src/lib.rs is intentionally empty so cargo treats the directory as a workspace member hosting integration tests under tests/.
• core/tests/physical_optimizer/mod.rs: drops the four moved mod declarations; adds dead_code suppression for test_utils (some helpers were only used by the moved tests).
• Shared test_utils: the moved files reference the existing helper via #[path = "../../core/tests/physical_optimizer/test_utils.rs"] mod test_utils; so there's a single source of truth. The mod declaration carries #[allow(dead_code, clippy::allow_attributes)] because each test binary uses only a subset of the helpers.
• enforce_sorting.rs: DummyStreamPartition (previously imported via crate::memory_limit::DummyStreamPartition) is inlined as a 20-line struct.
• enforce_sorting_monotonicity.rs: EnforceSortingTest (previously shared with enforce_sorting.rs as pub(crate)) is duplicated into this file — each integration test is now its own crate root and cannot reach into a sibling integration test's pub(crate) items. Annotated as transient.

Verification

cargo build --workspace --tests
cargo test -p datafusion-physical-optimizer-tests          # 236 passed (66 + 60 + 64 + 46)
cargo test -p datafusion --test core_integration physical_optimizer::   # remaining core tests still pass
cargo clippy --workspace --tests -- -D warnings
cargo fmt --all --check

cargo metadata confirms datafusion-physical-optimizer-tests is a leaf in the workspace dependency graph (no incoming edges), so dev/depcheck will not find a cycle through it.

Follow-ups

• Rebase Add EnsureRequirements: merged EnforceDistribution + EnforceSorting with idempotent pushdown_sorts #21976 on top of this PR; the diff there should be the function-name swap from EnforceDistribution / EnforceSorting to EnsureRequirements on the four moved files plus genuinely new tests.
• Move the remaining core/tests/physical_optimizer/ integration tests alongside their rules in a follow-up. The #[path] reference to core/tests/.../test_utils.rs will go away naturally when that happens.

[Copilot]

Pull request overview

This PR reorganizes physical optimizer integration tests by moving the existing EnforceDistribution / EnforceSorting test suites into the datafusion-physical-optimizer crate’s tests/ directory, so future work (notably EnsureRequirements in #21976) can show cleaner diffs focused on rule behavior rather than test relocation.

Changes:

• Relocates the four large optimizer rule integration test files into datafusion/physical-optimizer/tests/ and updates their imports to use a local tests/common/test_utils.rs.
• Adds datafusion-physical-optimizer dev-dependencies needed to compile/run these integration tests in the new crate.
• Adjusts datafusion/core/tests/physical_optimizer/mod.rs to stop compiling the moved test modules and to suppress now-dead test_utils helpers.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
datafusion/physical-optimizer/tests/enforce_distribution.rs	Switched to local tests/common/test_utils.rs and minor wiring tweaks after move.
datafusion/physical-optimizer/tests/enforce_sorting.rs	Inlines DummyStreamPartition and switches to local tests/common/test_utils.rs.
datafusion/physical-optimizer/tests/enforce_sorting_monotonicity.rs	Duplicates EnforceSortingTest helper and rewires imports for standalone integration test crate.
datafusion/physical-optimizer/tests/replace_with_order_preserving_variants.rs	Switches to local tests/common/test_utils.rs and applies file-level clippy expectation.
datafusion/physical-optimizer/tests/common/test_utils.rs	Adds shared optimizer test helpers for the moved integration tests.
datafusion/physical-optimizer/Cargo.toml	Adds dev-dependencies required by the moved integration tests.
datafusion/core/tests/physical_optimizer/mod.rs	Removes moved module declarations and adds temporary lint suppression for remaining helpers.
Cargo.lock	Updates lockfile due to new dev-dependencies.

Comments suppressed due to low confidence (1)

datafusion/physical-optimizer/tests/enforce_sorting_monotonicity.rs:52

• TransformedResult is imported but not used. This will trigger an unused-import warning and can fail CI when running clippy/rustc with -D warnings. Remove TransformedResult from the import list (keep TreeNode if it is needed for transform_up method resolution).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zhuqi-lucas]

cc @alamb — context: the original plan was to move these tests to datafusion/physical-optimizer/tests/, but that requires datafusion as a dev-dep of datafusion-physical-optimizer (the tests use SessionContext), which dev/depcheck rejects. Pivoted to a new leaf workspace crate datafusion-physical-optimizer-tests (publish = false) — see the "Why a new crate" section in the PR description for the full rationale. Happy to switch to a different structure if you prefer.

[alamb]

Pivoted to a new leaf workspace crate datafusion-physical-optimizer-tests (publish = false) — see the "Why a new crate" section in the PR description for the full rationale. Happy to switch to a different structure if you prefer.

How about just moving the tests to core/tests/physical_optimizer/ with some of the other physical optimizer tests that have the same property?

[alamb]

Hi @zhuqi-lucas -- I am sorry but I am really confused now.

What I was trying to get at on #21976 is that I think we should reduce the risk of the rewrite by reusing the existing tests for EnforceSorting and EnforceDistribution

I think the least risky way to avoid regressions is to reuse all the existing regression tests for those two passes (and make sure they pass with the new EnsureRequirements pass

However upon additional review it seems like what actually happened was that you disconnected EnsureSorting and EnsureDistribution from the optimizer passes but left them in the code base so the old tests are still present. Then you / your agent recreated a new copy of many of the tests just in terms of EnsureRequirements (both in the new modules theselves like datafusion/physical-optimizer/src/ensure_requirements/mod.rs as well as in other modules like datafusion/physical-optimizer/src/ensure_requirements/new_tests.rs)

Given all these new tests I was under the (incorrect) impression that you had also deleted the existing tests for EnsureSorting and EnsureDistribution

So I guess maybe we already have the tests consolidated in core/tests/physical_optimizer and what is left to do is

1. remove the old EnsureDistribution and EnsureSorting passes
2. Update the existing tests to us EnsureRequirements
3. Remove any redundant tests that were added in Add EnsureRequirements: merged EnforceDistribution + EnforceSorting with idempotent pushdown_sorts #21976

Thank you for pushing this along -- I think this is one of the most complicated parts of the physical optimizer (so it is the hardest parts to change, but also one of the most important things to reduce compleixty)