Skip to content

[SPARK-56874][SQL] Fast-path DateTimeUtils.daysToMicros for ZoneOffset.UTC #55893

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
LuciferYang wants to merge 10 commits into
base: master
Choose a base branch
from
Draft

[SPARK-56874][SQL] Fast-path DateTimeUtils.daysToMicros for ZoneOffset.UTC #55893

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
72a6bfa
[SPARK-56874][SQL] Fast-path DateTimeUtils.daysToMicros for ZoneOffse…
LuciferYang May 15, 2026
0e828a4
[SPARK-56874][SQL][FOLLOWUP] Apply scalafmt to daysToMicros docstring
LuciferYang May 15, 2026
647bcff
Benchmark results for org.apache.spark.sql.execution.datasources.parq…
LuciferYang May 15, 2026
798ac97
rebase
LuciferYang May 16, 2026
1ecd3bf
Benchmark results for org.apache.spark.sql.execution.datasources.parq…
LuciferYang May 16, 2026
bf34956
Benchmark results for org.apache.spark.sql.execution.datasources.parq…
LuciferYang May 16, 2026
eaaa8fe
Benchmark results for org.apache.spark.sql.execution.datasources.parq…
LuciferYang May 16, 2026
111df53
Benchmark results for org.apache.spark.sql.execution.datasources.parq…
LuciferYang May 16, 2026
4830073
Benchmark results for org.apache.spark.sql.execution.datasources.parq…
LuciferYang May 16, 2026
ef4391c
Benchmark results for org.apache.spark.sql.execution.datasources.parq…
LuciferYang May 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,11 +228,16 @@ trait SparkDateTimeUtils {

/**
* Converts days since 1970-01-01 at the given zone ID to microseconds since 1970-01-01
* 00:00:00Z.
* 00:00:00Z. When `zoneId eq ZoneOffset.UTC`, takes a direct-multiply fast path that skips the
* `LocalDate`/`ZonedDateTime`/`Instant` chain.
*/
def daysToMicros(days: Int, zoneId: ZoneId): Long = {
val instant = daysToLocalDate(days).atStartOfDay(zoneId).toInstant
instantToMicros(instant)
if (zoneId eq ZoneOffset.UTC) {
Copy link
Copy Markdown

@acruise acruise May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of curiosity, is there a version of this that takes an arbitrary offset (e.g. in seconds) vs. a nominal zone ID? The distinction hardly ever matters, except when various jurisdictions change their timezone rules. For example, here in British Columbia, we have officially stopped doing DST changes! 😅

Math.multiplyExact(days.toLong, MICROS_PER_DAY)
} else {
val instant = daysToLocalDate(days).atStartOfDay(zoneId).toInstant
instantToMicros(instant)
}
}

/**
Expand Down
26 changes: 26 additions & 0 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -971,6 +971,32 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
assert(DateTimeUtils.microsToMillis(-157700927876544L) === -157700927877L)
}

test("daysToMicros: ZoneOffset.UTC fast path matches the generic zone path") {
// The UTC fast path returns `days * MICROS_PER_DAY` directly; assert it agrees with the
// `LocalDate -> ZonedDateTime -> Instant` path used for any other zone whose offset is 0
// (e.g. `Etc/GMT`). Covers zero, positive, negative, and values bounded by the largest
// `days` for which `days * MICROS_PER_DAY` does not overflow `Long`.
val maxSafeDays = (Long.MaxValue / MICROS_PER_DAY).toInt
val cases = Seq(0, 1, -1, 365, -365, 16800, -16800, 1_000_000, -1_000_000,
maxSafeDays, -maxSafeDays)
val gmt = ZoneId.of("Etc/GMT")
cases.foreach { d =>
assert(daysToMicros(d, ZoneOffset.UTC) === daysToMicros(d, gmt),
s"UTC fast path diverged from Etc/GMT path at days=$d")
assert(daysToMicros(d, ZoneOffset.UTC) === d.toLong * MICROS_PER_DAY,
s"UTC fast path != days * MICROS_PER_DAY at days=$d")
}

// Overflow: any `days` past `maxSafeDays` overflows `Long` and must throw rather than
// silently wrap.
intercept[ArithmeticException] {
daysToMicros(maxSafeDays + 1, ZoneOffset.UTC)
}
intercept[ArithmeticException] {
daysToMicros(-maxSafeDays - 1, ZoneOffset.UTC)
}
}

test("SPARK-29012: special timestamp values") {
testSpecialDatetimeValues { zoneId =>
val tolerance = TimeUnit.SECONDS.toMicros(30)
Expand Down
52 changes: 26 additions & 26 deletions sql/core/benchmarks/ParquetVectorUpdaterBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,83 +2,83 @@
Identity Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
AMD EPYC 7763 64-Core Processor
Identity Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BooleanUpdater 0 0 0 17004.4 0.1 1.0X
ByteUpdater (INT32 -> Byte) 0 0 0 3748.2 0.3 0.2X
ShortUpdater (INT32 -> Short) 1 1 0 1683.0 0.6 0.1X
IntegerUpdater 0 0 0 10179.2 0.1 0.6X
LongUpdater 0 0 0 5075.5 0.2 0.3X
FloatUpdater 0 0 0 10201.9 0.1 0.6X
DoubleUpdater 0 0 0 5140.6 0.2 0.3X
BinaryUpdater 15 15 0 71.3 14.0 0.0X
ByteUpdater (INT32 -> Byte) 0 0 0 3746.5 0.3 0.2X
ShortUpdater (INT32 -> Short) 1 1 0 1681.2 0.6 0.1X
IntegerUpdater 0 0 0 10290.1 0.1 0.6X
LongUpdater 0 0 0 3875.9 0.3 0.2X
FloatUpdater 0 0 0 10148.5 0.1 0.6X
DoubleUpdater 0 0 0 5141.3 0.2 0.3X
BinaryUpdater 15 15 0 70.7 14.1 0.0X


================================================================================================
Type-converting Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
AMD EPYC 7763 64-Core Processor
Type-converting Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
IntegerToLongUpdater 0 0 0 6208.4 0.2 1.0X
IntegerToDoubleUpdater 0 0 0 6155.9 0.2 1.0X
FloatToDoubleUpdater 0 0 0 2526.5 0.4 0.4X
DateToTimestampNTZUpdater 27 28 1 38.6 25.9 0.0X
DowncastLongUpdater (INT64 -> Decimal(9,2)) 0 0 0 5828.5 0.2 0.9X
IntegerToLongUpdater 0 0 0 6237.3 0.2 1.0X
IntegerToDoubleUpdater 0 0 0 6117.3 0.2 1.0X
FloatToDoubleUpdater 0 0 0 2526.7 0.4 0.4X
DateToTimestampNTZUpdater 3 3 0 366.1 2.7 0.1X
DowncastLongUpdater (INT64 -> Decimal(9,2)) 0 0 0 5126.2 0.2 0.8X


================================================================================================
Rebase Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
AMD EPYC 7763 64-Core Processor
Rebase Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------
IntegerWithRebaseUpdater (DATE legacy) 0 0 0 3651.5 0.3 1.0X
LongWithRebaseUpdater (TIMESTAMP_MICROS legacy) 0 0 0 2621.2 0.4 0.7X
IntegerWithRebaseUpdater (DATE legacy) 0 0 0 3640.1 0.3 1.0X
LongWithRebaseUpdater (TIMESTAMP_MICROS legacy) 0 0 0 2281.8 0.4 0.6X
LongAsMicrosUpdater (TIMESTAMP_MILLIS) 2 3 0 420.5 2.4 0.1X


================================================================================================
Unsigned Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
AMD EPYC 7763 64-Core Processor
Unsigned Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------
UnsignedIntegerUpdater (UINT32 -> Long) 0 0 0 5879.9 0.2 1.0X
UnsignedIntegerUpdater (UINT32 -> Long) 0 0 0 5141.0 0.2 1.0X
UnsignedLongUpdater (UINT64 -> Decimal(20,0)) 16 17 0 63.8 15.7 0.0X


================================================================================================
Decimal Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
AMD EPYC 7763 64-Core Processor
Decimal Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
IntegerToDecimalUpdater 0 0 0 10144.6 0.1 1.0X
LongToDecimalUpdater 0 0 0 5061.5 0.2 0.5X
FixedLenByteArrayToDecimalUpdater 20 21 1 51.3 19.5 0.0X
IntegerToDecimalUpdater 0 0 0 10288.1 0.1 1.0X
LongToDecimalUpdater 0 0 0 3872.9 0.3 0.4X
FixedLenByteArrayToDecimalUpdater 21 21 1 50.2 19.9 0.0X


================================================================================================
FixedLenByteArray Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
AMD EPYC 7763 64-Core Processor
FixedLenByteArray Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------------
FixedLenByteArrayUpdater (len=16 -> Binary) 20 21 1 51.8 19.3 1.0X
FixedLenByteArrayUpdater (len=16 -> Binary) 20 21 2 51.8 19.3 1.0X
FixedLenByteArrayAsIntUpdater (len=4 -> Decimal(9,2)) 7 7 0 160.2 6.2 3.1X
FixedLenByteArrayAsLongUpdater (len=8 -> Decimal(18,4)) 8 8 0 133.2 7.5 2.6X
FixedLenByteArrayAsLongUpdater (len=8 -> Decimal(18,4)) 8 8 0 133.3 7.5 2.6X


Loading