[SPARK-56867][SQL] Respect spark.jars.ivySettings when downloading Hive metastore jars

[sarutak]

What changes were proposed in this pull request?

This PR modifies IsolatedClientLoader.downloadVersion to respect the spark.jars.ivySettings configuration when resolving Hive metastore jars via spark.sql.hive.metastore.jars=maven.

Previously, downloadVersion always called MavenUtils.buildIvySettings directly, ignoring any custom Ivy settings file. With this change, if spark.jars.ivySettings is set, MavenUtils.loadIvySettings is used instead, consistent with how DependencyUtils.resolveMavenDependencies handles --packages.

Why are the changes needed?

When spark.sql.hive.metastore.jars=maven is configured, Spark downloads Hive metastore jars using Apache Ivy. However, unlike --packages / spark.jars.packages, this code path did not honor spark.jars.ivySettings. This made it impossible to download Hive metastore jars from authenticated private repositories, since credentials can only be configured through an Ivy settings file.

Does this PR introduce any user-facing change?

Yes. Users who set spark.jars.ivySettings can now have that configuration apply to Hive metastore jar resolution as well. Previously, the Ivy settings file was only used for spark.jars.packages.

How was this patch tested?

Confirmed it works with a private repository which requires authentication.

Was this patch authored or co-authored using generative AI tooling?

Kiro CLI / Opus 4.7

[dongjoon-hyun]

Do you think we can have a test coverage for this to prevent a future regression, @sarutak ?