feat(isthmus): configurable fallback to dynamic function mapping

[ZorinAnton]

This PR introduces a new autoFallbackToDynamicFunctionMapping feature flag that enables automatic support for all unmapped extension functions (scalar, aggregate, and window functions) in SQL↔Substrait conversions.

[vbarua]

I'm working on a PR (#649) to try and make wiring in what you're doing here easier. We're starting to run into the limits of the existing design of the function conversion system, and I'm wary of adding more complexity before cleaning it up.

My big issue is with how we're using the feature flags for this, because we're embedding a ton of conditional configuration logic into constructors all over the codebase, which makes it very difficult to reason about an already difficult part of the system. What I would like to do is separate out the configuration from the constructors. Effectively, can we construct all the function converters we need outside of the SubstraitRelNode and SubstraitRelVisitor, and then inject them in.

[nielspardon]

@ZorinAnton can you rebase one more time and resolve the conflicts? thanks

[vbarua]

High-level question

What's the value of having a separate AutomaticDynamicFunctionMappingConverterProvider instead of expanding the DynamicConverterProvider. Put another way, would it make sense to just have a non-dynamic mode (ConverterProvider) and dynamic mode (DynamicConverterProvider)?

[ZorinAnton]

While they use similar mechanics under the hood, they serve two distinct use cases:

• DynamicConverterProvider is specifically for custom UDFs injected at runtime. It explicitly filters out the standard Substrait catalog.
• AutomaticDynamicFunctionMappingConverterProvider is a fallback for standard Substrait functions (like strftime) that exist in the official YAMLs but haven't been manually added to our static FunctionMappings.java yet.

Keeping them separate allows users to opt into custom UDF support, the standard function fallback, or both independently.

[bestbeforetoday]

With the current ConverterProvider structure, it is not clear to me how we expect users to opt in to both custom UDF and standard function fallback. Maybe we need some thought on composable ConverterProvider implementations. However, I think any changes to support that should happen in another PR. This one has already been pending for too long.

[bestbeforetoday]

@ZorinAnton I had some concerns about:

1. The merge commits instead of rebasing of your changes on the latest main branch state, which made the commit history hard to unpick.
2. Some implementation details: readability, complexity and fragility.

I have cherry-picked your commits on top of the current main branch content to tidy up the commit history. I also added some additional commits, just to tidy up the implementation slightly. These changes address things I would otherwise have raised as review comments, but I have made the changes myself in the interests of getting this PR merged quickly. The logic and behaviour are unchanged. This is pushed to your PR branch so this PR reflects all of these changes.

Your exact PR branch state before the changes I made above is at bestbeforetoday/anton-fallback-to-dynamic-function-mappings.

I am happy with this PR to be merged as-is so am approving. Another pair of eyes — either yours or another maintainer — might be good as a sanity check before merging.

[ZorinAnton]

With the current ConverterProvider structure, it is not clear to me how we expect users to opt in to both custom UDF and standard function fallback.

Potentially, the Composite Pattern can be used. This might cause creation of new class for each combination of features, but at least users could opt in excaclty to what they want.

Another pair of eyes — either yours or another maintainer — might be good as a sanity check before merging.

At first glance I don't any issues, I will take a deeper look over the weekend.