Skip to content

deps: test DataFusion 53#3629

Closed
mbutrovich wants to merge 9 commits intoapache:mainfrom
mbutrovich:df53
Closed

deps: test DataFusion 53#3629
mbutrovich wants to merge 9 commits intoapache:mainfrom
mbutrovich:df53

Conversation

@mbutrovich
Copy link
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

Dependency changes:

  • datafusion 52.2.0 → git branch-53 (53.0.0)
  • datafusion-datasource, datafusion-physical-expr-adapter, datafusion-spark → git branch-53
  • datafusion-spark now with features = ["core"] (workaround for branch-53 bug)
  • arrow 57.3.0 → 58.0.0
  • parquet 57.3.0 → 58.0.0
  • object_store 0.12.3 → 0.13.1

API fixes:

  • ExecutionPlan::properties() returns &Arc<PlanProperties> — wrapped cache fields in Arc across 5 files
  • Removed ExecutionPlan::statistics() from parquet_writer and shuffle_writer
  • HashJoinExec::try_new takes new null_aware: bool param — added false
  • PhysicalExprAdapterFactory::create now returns Result — added ? and Ok()
  • EncryptionFactory methods return Result<...> instead of Result<..., DataFusionError>
  • DefaultPhysicalExprAdapterFactory::create now returns Result — added ?

Feature-gated (blocked on upstream):

  • iceberg — gated behind iceberg feature (iceberg-rust still on arrow 57)
  • hdfs-opendal — removed from defaults (object_store_opendal still on 0.12)
  • hdfs crate needs object_store 0.13 migration (non-default member, deferred)

Remaining warnings:

  • CoalesceBatchesExec is deprecated (needs separate migration)
  • runtime_env unused when hdfs-opendal is off (expected)

How are these changes tested?

Existing tests.

@mbutrovich mbutrovich closed this Mar 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant