fix(backend/kernel): comparator parity — async retention + intervals_as_string + precision/scale + named params + utils.exc fix

Five small comparator-parity fixes surfaced by the same audit run.
All are independently small but ship together because they share
the kernel-backend client surface and the audit baseline.

### 1. Async Statement retention

`KernelDatabricksClient.execute_command` closed the parent `Statement`
in `finally` regardless of `async_op`. The kernel's `Statement.close()`
invalidates child handles (see databricks-sql-kernel
`src/statement/validity.rs`), so the async handle died before the user
could poll it — breaking `execute_async` → `is_query_pending` →
`get_async_execution_result`.

Fix: when `async_op=True`, retain the parent Statement in a new
`_async_statements` dict alongside `_async_handles`, and close it
from `close_command`, `close_session`, and `get_execution_result`
after the executed handle is done.

Comparator outcome: STATEMENT_ASYNC suite 3/3 match (was 0/3).

### 2. `intervals_as_string` wire-through

pyarrow's Python bindings cannot decode Arrow's `month_interval` type
(id 21 — `KeyError` from `.as_py`, `to_pylist`, `cast(string)`,
`to_pandas`). Every kernel-backend `SELECT *` over a table with an
`INTERVAL YEAR TO MONTH` column raised `ArrowNotImplementedError` —
32 / 88 audit diffs.

Fix: pass `intervals_as_string=True` to the kernel `Session(...)`
constructor unconditionally. The kernel post-processor stringifies
`Interval` / `Duration` columns server-side to `Utf8` (kernel PR #64).

Comparator outcome: bucket A (ArrowNotImplementedError) 32 → 0.

### 3. Decimal precision/scale extraction

`description_from_arrow_schema` hard-coded `None` for PEP 249
description-tuple slots 4 and 5. For DECIMAL columns the Arrow
schema carries `precision` / `scale` on `Decimal128Type`, but they
were silently dropped — 88 cell diffs (44 precision + 44 scale).

Fix: factor out `_precision_scale_for_arrow_type(arrow_type)` and
call it from the description builder. Today it only handles
decimals; future extensions slot in here.

Comparator outcome: 88 precision+scale diffs → 0.

### 4. Named-parameter binding

`bind_tspark_params` raised `NotSupportedError` for any
`TSparkParameter` with `name` set. The canonical SEA proto marks
`StatementParameter.name` as `openapi_required=true` (named is the
spec-required public form; `ordinal` is `PUBLIC_UNDOCUMENTED`).
Kernel PR #65 added a `Statement.bind_named_param` PyO3 API.

Fix: route named bindings via the new API. Positional ordinals
self-increment so a named entry in the middle of the list doesn't
claim a positional slot.

Comparator outcome: PREPARED_STATEMENT_NAMED 1/1 match (was 0/1).
Full thrift-vs-kernel run: 97/132 match (was 96/132).

### 5. `utils.ParamEscaper` `exc.ProgrammingError` import fix

`ParamEscaper.escape_args` and `escape_item` both raised
`exc.ProgrammingError(...)`, but `exc` was never imported. On any
unsupported parameter shape the user saw `NameError: name 'exc' is
not defined` instead of a clean PEP 249 `ProgrammingError`.

Surfaced via the same audit harness running INLINE_PARAMS: both
backends raised `NameError`, which the comparator's class+message
match treated as parity — a false-positive that hid both the driver
bug and the underlying caller-side type mismatch.

Fix: import `ProgrammingError` directly from `databricks.sql.exc`
(matching the pattern used in `session.py`, `client.py`,
`result_set.py`, etc.) and replace the two `exc.ProgrammingError(...)`
sites with bare `ProgrammingError(...)`.

## Dependencies

Items #2, #3, and #4 require the matching databricks-sql-kernel
changes: PR #64 (`intervals_as_string` + empty-result schema fix)
and PR #65 (named-param binding). For local testing the comparator
harness uses `KERNEL_FREEZE=1` against a kernel checkout of the
feature branches.

## Headline comparator results (full thrift-vs-kernel run)

|                                    | Before | After |
|------------------------------------|--------|-------|
| match / diff / skipped             | 60 / 88 / 0 | **97 / 34 / 1** |
| Bucket A (ArrowNotImplementedError)| 32     | 0     |
| `decimal_column` precision/scale   | 88     | 0     |
| Bucket B1 (named params)           | 1      | 0     |
| Suites fully clean                 | 12 / 30 | **17 / 30** |

Remaining 34 diffs cluster into documented causes (complex types in
`fetchall_arrow`, timestamp tz on Arrow path, VOID surface,
METADATA pattern semantics) — tracked in
`~/docs/python-kernel/comparator-diff-tasklist.md`.

## Test plan

- [x] Manual repro of async path: `cur.execute_async("SELECT 1")` →
      `is_query_pending` → `get_async_execution_result` → `fetchall`
      succeeds on `use_kernel=True`
- [x] Manual repro of interval path: `cur.execute("SELECT ym_interval_column FROM ...")`
      returns string-shaped rows matching the Thrift surface
- [x] Manual repro of decimal path: `cur.description` for a
      `DECIMAL(10,2)` column now reports `precision=10, scale=2`
- [x] Live e2e: `test_parameterized_query_named_params` /
      `test_parameterized_query_named_param_with_null` (added)
- [x] Unit tests: `bind_tspark_params` named/positional/mixed
      routing (`tests/unit/test_kernel_type_mapping.py`)
- [x] Manual repro of utils fix:
      `ParamEscaper().escape_args(object())` raises
      `ProgrammingError`, not `NameError`
- [x] Full thrift-vs-kernel comparator: 97 / 34 / 1 of 132
- [x] Existing connector unit suite: 752 passed

Co-authored-by: Isaac

Author: vikrantpuppala

src/databricks/sql/backend/kernel/client.py

@@ -129,6 +129,15 @@ def __init__(
         # Guarded by ``_async_handles_lock`` so concurrent cursors on the
         # same connection don't race on submit / close / close-session.
         self._async_handles: Dict[str, Any] = {}
+        # Parent ``Statement`` objects kept alive alongside async handles.
+        # On the kernel, ``Statement.close()`` flips the validity flag on
+        # the produced executed handle (see kernel
+        # ``statement::mutable::close``), so we cannot close the
+        # Statement immediately after ``submit()`` as we do for sync
+        # ``execute()``. Instead retain it here and close it in
+        # ``close_command`` / ``close_session`` after the async handle
+        # has finished its work.
+        self._async_statements: Dict[str, Any] = {}
         # CommandId.guids of async commands that have already been
         # closed (via ``close_command`` or ``close_session``). Lets
         # ``get_query_state`` report ``CLOSED`` for them rather than
@@ -167,6 +176,16 @@ def open_session(
                 schema=schema or self._schema,
                 session_conf=session_conf,
                 complex_types_as_json=not self._use_arrow_native_complex_types,
+                # Pyarrow's Python bindings cannot decode Arrow's
+                # ``month_interval`` type at all (id 21 — raises
+                # ``KeyError`` from ``.as_py``, ``to_pylist``,
+                # ``cast(string)``, and ``to_pandas``). Ask the kernel
+                # to stringify INTERVAL / DURATION columns server-side
+                # so result sets containing interval columns are
+                # decodable on the Python side. Matches the Thrift
+                # backend's surface (interval columns arrive as
+                # strings).
+                intervals_as_string=True,
                 **auth_kwargs,
             )
         except Exception as exc:
@@ -197,7 +216,9 @@ def close_session(self, session_id: SessionId) -> None:
         # server-side CloseStatement before the session goes away.
         with self._async_handles_lock:
             tracked = list(self._async_handles.items())
+            tracked_stmts = list(self._async_statements.items())
             self._async_handles.clear()
+            self._async_statements.clear()
             for guid, _ in tracked:
                 self._closed_commands.add(guid)
         for _, handle in tracked:
@@ -211,6 +232,16 @@ def close_session(self, session_id: SessionId) -> None:
                 logger.warning(
                     "Error closing async handle during session close: %s", exc
                 )
+        # Now drop the parent Statements that were keeping those handles
+        # alive. Same non-fatal close semantics — close errors are not
+        # actionable at session-close time.
+        for _, stmt in tracked_stmts:
+            try:
+                stmt.close()
+            except Exception as exc:
+                logger.warning(
+                    "Error closing async statement during session close: %s", exc
+                )
         try:
             self._kernel_session.close()
         except Exception as exc:
@@ -249,6 +280,11 @@ def execute_command(
             stmt = self._kernel_session.statement()
         except Exception as exc:
             raise _wrap_kernel_exception("execute_command", exc) from exc
+        # ``async_op`` keeps ``stmt`` alive (tracked in
+        # ``_async_statements`` and closed by ``close_command``); the sync
+        # path drops it in finally. ``close_stmt`` is the post-success
+        # decision flag — it stays True on sync, flips to False on async.
+        close_stmt = True
         try:
             try:
                 stmt.set_sql(operation)
@@ -262,21 +298,26 @@ def execute_command(
                     cursor.active_command_id = command_id
                     with self._async_handles_lock:
                         self._async_handles[command_id.guid] = async_exec
+                        # Closing the kernel ``Statement`` invalidates the
+                        # async handle (see kernel validity flag). Retain
+                        # the Statement here and close it on
+                        # ``close_command`` / ``close_session``.
+                        self._async_statements[command_id.guid] = stmt
+                    close_stmt = False
                     return None
                 executed = stmt.execute()
             except Exception as exc:
                 raise _wrap_kernel_exception("execute_command", exc) from exc
         finally:
-            # ``Statement`` is a lifecycle owner separate from the
-            # executed handle it produces. Drop it here so the
-            # parent doesn't keep the handle alive longer than the
-            # caller expects. Swallow all close errors (including
-            # PyO3 native exceptions) — a failed stmt.close() is
-            # not actionable for the caller.
-            try:
-                stmt.close()
-            except Exception:
-                pass
+            if close_stmt:
+                # Sync path: ``Statement`` is a lifecycle owner separate
+                # from the executed handle. Drop it here so the parent
+                # doesn't outlive its caller. Swallow close errors —
+                # they're not actionable.
+                try:
+                    stmt.close()
+                except Exception:
+                    pass
 
         command_id = CommandId.from_sea_statement_id(executed.statement_id)
         cursor.active_command_id = command_id
@@ -307,17 +348,34 @@ def cancel_command(self, command_id: CommandId) -> None:
     def close_command(self, command_id: CommandId) -> None:
         with self._async_handles_lock:
             handle = self._async_handles.pop(command_id.guid, None)
+            stmt = self._async_statements.pop(command_id.guid, None)
             if handle is not None:
                 # Record the close so ``get_query_state`` can report
                 # ``CLOSED`` (not ``SUCCEEDED``) for this command.
                 self._closed_commands.add(command_id.guid)
         if handle is None:
             logger.debug("close_command: no tracked handle for %s", command_id)
+            # Still drop the parent Statement if somehow tracked without
+            # the handle — keeps the invariant clean even on bookkeeping
+            # races.
+            if stmt is not None:
+                try:
+                    stmt.close()
+                except Exception:
+                    pass
             return
         try:
             handle.close()
         except Exception as exc:
             raise _wrap_kernel_exception("close_command", exc) from exc
+        finally:
+            # Now safe to close the parent Statement — the executed
+            # handle has finished its lifecycle.
+            if stmt is not None:
+                try:
+                    stmt.close()
+                except Exception:
+                    pass
 
     def get_query_state(self, command_id: CommandId) -> CommandState:
         with self._async_handles_lock:
@@ -378,6 +436,7 @@ def get_execution_result(
         # it wraps. Drop tracking and fire-and-forget the close.
         with self._async_handles_lock:
             self._async_handles.pop(command_id.guid, None)
+            stmt = self._async_statements.pop(command_id.guid, None)
             self._closed_commands.add(command_id.guid)
         try:
             async_exec.close()
@@ -387,6 +446,18 @@ def get_execution_result(
                 command_id,
                 exc,
             )
+        # The parent Statement is no longer needed once the async handle
+        # has produced its ResultStream. Close to release server-side
+        # tracking; matches the sync path's eager Statement close.
+        if stmt is not None:
+            try:
+                stmt.close()
+            except Exception as exc:
+                logger.warning(
+                    "Error closing async statement after await_result for %s: %s",
+                    command_id,
+                    exc,
+                )
         # ``KernelResultSet.__init__`` calls ``arrow_schema()`` which
         # can raise — map that to PEP 249 too.
         try:

src/databricks/sql/backend/kernel/type_mapping.py

@@ -21,7 +21,7 @@
 
 from __future__ import annotations
 
-from typing import Any, List, Tuple
+from typing import Any, List, Optional, Tuple
 
 import pyarrow
 
@@ -102,21 +102,47 @@ def description_from_arrow_schema(schema: pyarrow.Schema) -> List[Tuple]:
     backend's behaviour; other precise types (``INTERVAL_*``,
     ``GEOMETRY``, ``GEOGRAPHY``) collapse to their Arrow shape on
     both backends and don't need a remap.
+
+    ``precision`` / ``scale`` are extracted from ``Decimal128Type`` /
+    ``Decimal256Type`` so DECIMAL columns expose the same
+    ``(precision, scale)`` pair the Thrift backend reports. The Arrow
+    schema carries these on the type itself; without this extraction
+    the kernel-backend description would silently drop them, breaking
+    parity for any consumer (SQLAlchemy, pandas-read-sql, etc.) that
+    reads slots 4/5 to know how to display or round decimal values.
     """
     return [
         (
             field.name,
             _databricks_type_for_field(field),
             None,
             None,
-            None,
-            None,
+            *_precision_scale_for_arrow_type(field.type),
             None,
         )
         for field in schema
     ]
 
 
+def _precision_scale_for_arrow_type(
+    arrow_type: pyarrow.DataType,
+) -> Tuple[Optional[int], Optional[int]]:
+    """Extract PEP 249 ``(precision, scale)`` from an Arrow type.
+
+    Only Arrow's decimal types carry both; every other type collapses
+    to ``(None, None)`` to match the Thrift backend's behaviour. Future
+    extensions (e.g. fractional-second precision from
+    ``Time64Type`` / ``Timestamp``) can land here without touching the
+    description builder above.
+    """
+    if pyarrow.types.is_decimal(arrow_type):
+        # Decimal128Type / Decimal256Type both expose `.precision` and
+        # `.scale`. The cast is for the type checker — pyarrow's
+        # `DataType` base type doesn't declare them.
+        return arrow_type.precision, arrow_type.scale  # type: ignore[attr-defined]
+    return None, None
+
+
 def _databricks_type_for_field(field: pyarrow.Field) -> str:
     """Pick the PEP 249 type code for a single field.
 
@@ -173,32 +199,19 @@ def _tspark_param_value_str(param: ttypes.TSparkParameter) -> Any:
 def bind_tspark_params(kernel_stmt, parameters: List[ttypes.TSparkParameter]) -> None:
     """Bind a list of ``TSparkParameter`` onto a kernel ``Statement``.
 
-    The kernel expects positional bindings only (SEA v0 doesn't
-    accept named bindings on the wire). The connector's
+    Both positional and named bindings are supported. The connector's
     ``TSparkParameter`` has an ``ordinal: bool`` flag; ``True`` means
-    "treat as positional in source-list order". Named-binding
-    parameters surface as ``NotSupportedError`` so the user gets a
-    clear message instead of a server-side rejection.
+    "treat as positional in source-list order", otherwise the
+    parameter is bound by name via ``Statement.bind_named_param``.
 
     Compound types (``ARRAY`` / ``MAP`` / ``STRUCT``) build a
     ``TSparkParameter`` with the payload on ``arguments`` and
     ``value=None`` — forwarding that would silently bind a typed
     NULL. Reject up front with ``NotSupportedError`` so callers get
     a clear message instead of silent data loss.
     """
-    for i, param in enumerate(parameters, start=1):
-        # ``ordinal`` on connector-native params is a bool (True for
-        # positional, False for named). Thrift defaults to ``None``;
-        # treat any non-True value with a name as a named binding so
-        # a future caller that forgets to set ordinal=True still gets
-        # rejected instead of silently dropping the name.
-        name = getattr(param, "name", None)
-        if name and getattr(param, "ordinal", None) is not True:
-            raise NotSupportedError(
-                f"Named parameter binding (got name={name!r}) is not yet "
-                "supported on the kernel backend; pass parameters positionally."
-            )
-
+    positional_index = 0
+    for param in parameters:
         sql_type = param.type or "STRING"
         # Compound types put their payload on ``arguments``, not
         # ``value``. The kernel parser doesn't accept them yet, and
@@ -214,7 +227,12 @@ def bind_tspark_params(kernel_stmt, parameters: List[ttypes.TSparkParameter]) ->
             )
 
         value_str = _tspark_param_value_str(param)
-        # The kernel takes 1-based ordinals; `i` is already that.
-        # Errors from the kernel side (bad literal, unsupported type,
-        # etc.) come up as KernelError and bubble through normally.
-        kernel_stmt.bind_param(i, value_str, sql_type)
+        # ``ordinal`` on connector-native params is a bool. ``True``
+        # → positional (assign the next 1-based ordinal). Anything
+        # else with a name → named binding.
+        name = getattr(param, "name", None)
+        if name and getattr(param, "ordinal", None) is not True:
+            kernel_stmt.bind_named_param(name, value_str, sql_type)
+        else:
+            positional_index += 1
+            kernel_stmt.bind_param(positional_index, value_str, sql_type)

src/databricks/sql/utils.py

@@ -19,6 +19,7 @@
     pyarrow = None
 
 from databricks.sql import OperationalError
+from databricks.sql.exc import ProgrammingError
 from databricks.sql.cloudfetch.download_manager import ResultFileDownloadManager
 from databricks.sql.thrift_api.TCLIService.ttypes import (
     TRowSet,
@@ -548,7 +549,7 @@ def escape_args(self, parameters):
         elif isinstance(parameters, (list, tuple)):
             return tuple(self.escape_item(x) for x in parameters)
         else:
-            raise exc.ProgrammingError(
+            raise ProgrammingError(
                 "Unsupported param format: {}".format(parameters)
             )
 
@@ -606,7 +607,7 @@ def escape_item(self, item):
         elif isinstance(item, Mapping):
             return self.escape_mapping(item)
         else:
-            raise exc.ProgrammingError("Unsupported object {}".format(item))
+            raise ProgrammingError("Unsupported object {}".format(item))
 
 
 def inject_parameters(operation: str, parameters: Dict[str, str]):

tests/e2e/test_kernel_backend.py

@@ -241,6 +241,34 @@ def test_parameterized_query_with_null(conn):
         assert rows[0][0] is True
 
 
+def test_parameterized_query_named_params(conn):
+    """Named parameter binding via the kernel backend. The
+    connector passes ``parameters={name: value}`` dicts (DB-API
+    style); the kernel forwards them through ``bind_named_param``
+    so the SEA wire payload sets ``StatementParameter.name`` (the
+    spec-required public form per canonical proto).
+    """
+    with conn.cursor() as cur:
+        cur.execute(
+            "SELECT :n AS n, :s AS s, :b AS b",
+            {"n": 42, "s": "alice", "b": True},
+        )
+        rows = cur.fetchall()
+        assert len(rows) == 1
+        assert rows[0][0] == 42
+        assert rows[0][1] == "alice"
+        assert rows[0][2] is True
+
+
+def test_parameterized_query_named_param_with_null(conn):
+    """``None`` value in a named binding flows through as
+    VoidParameter → kernel ``TypedValue::Null``."""
+    with conn.cursor() as cur:
+        cur.execute("SELECT :x IS NULL AS is_null", {"x": None})
+        rows = cur.fetchall()
+        assert rows[0][0] is True
+
+
 def test_parameterized_query_decimal(conn):
     """DECIMAL parameters carry precision/scale in the SQL type
     string ('DECIMAL(p,s)') — the kernel parser extracts them so