Skip to content

fix: convert JSON content blocks to text for Amazon Nova models in tool results#1891

Open
giulio-leone wants to merge 1 commit intostrands-agents:mainfrom
giulio-leone:fix/nova-json-tool-result
Open

fix: convert JSON content blocks to text for Amazon Nova models in tool results#1891
giulio-leone wants to merge 1 commit intostrands-agents:mainfrom
giulio-leone:fix/nova-json-tool-result

Conversation

@giulio-leone
Copy link
Contributor

Summary

Fixes #1095

Amazon Nova models (Pro, Lite, Micro) hallucinate when tool results contain JSON content blocks ({"json": {...}}). The models produce incoherent responses, fabricate information, role-play both sides of conversations, and hallucinate fake XML tool calls. Claude models handle JSON content blocks correctly.

Root Cause

The _format_request_message_content() method passes JSON content blocks through to the Bedrock API as-is:

if "json" in tool_result_content:
    formatted_content.append({"json": tool_result_content["json"]})

While this is valid per the Bedrock API spec, Nova models cannot process JSON content blocks correctly and produce hallucinated output instead.

Fix

Added model-aware JSON-to-text conversion that automatically serializes JSON content blocks to their text representation (json.dumps()) when the model is an Amazon Nova variant. Non-Nova models continue to receive JSON blocks unchanged.

The implementation follows the existing pattern used for tool result status handling (_MODELS_INCLUDE_STATUS / _should_include_tool_result_status()), adding a parallel _MODELS_CONVERT_JSON_TO_TEXT list and _should_convert_json_to_text() method.

Before (Nova receives JSON block → hallucination)

{"toolResult": {"content": [{"json": {"key": "value"}}], "toolUseId": "..."}}

After (Nova receives text → correct behavior)

{"toolResult": {"content": [{"text": "{\"key\": \"value\"}"}], "toolUseId": "..."}}

Changes

  • src/strands/models/bedrock.py:
    • Added _MODELS_CONVERT_JSON_TO_TEXT list with "amazon.nova" prefix
    • Added _should_convert_json_to_text() helper method
    • Updated JSON block handling in _format_request_message_content() to convert to text for Nova models

Testing

Added 5 new tests:

  • test_nova_model_converts_json_to_text_in_tool_result — basic JSON-to-text conversion
  • test_nova_model_converts_mixed_json_and_text_in_tool_result — mixed content preservation
  • test_claude_model_preserves_json_in_tool_result — non-Nova models unaffected
  • test_nova_model_handles_nested_json_in_tool_result — deeply nested structures
  • test_should_convert_json_to_text_nova_variants — all Nova model ID patterns

All 1871 existing + new tests pass.

@giulio-leone
Copy link
Contributor Author

Friendly ping — converts JSON content blocks to text format for Amazon Nova models in tool results, which don't support structured content in tool responses.

@giulio-leone
Copy link
Contributor Author

Rebased this branch onto the latest main and force-pushed it.

Local validation I ran:

  • .venv/bin/python -m pytest tests/strands/models/test_bedrock.py -k 'nova_model_converts_json_to_text_in_tool_result or nova_model_converts_mixed_json_and_text_in_tool_result or claude_model_preserves_json_in_tool_result or nova_model_handles_nested_json_in_tool_result or should_convert_json_to_text_nova_variants' -q -> 5 passed\n\nReal branch-vs-main proof (repo venv + explicit PYTHONPATH=<checkout>/src to verify which checkout was imported):\n- main imported sdk-python-main/src/strands/models/bedrock.py\n - Nova: has_convert_helper=False, content={"json": {"key": "value", "nested": {"n": 42}}}\n - Claude: has_convert_helper=False, content={"json": {"key": "value", "nested": {"n": 42}}}\n- this branch imported sdk-python/src/strands/models/bedrock.py\n - Nova: has_convert_helper=True, convert_json_to_text=True, content={"text": "{\\"key\\": \\"value\\", \\"nested\\": {\\"n\\": 42}}"}\n - Claude: has_convert_helper=True, convert_json_to_text=False, content={"json": {"key": "value", "nested": {"n": 42}}}\n\nSo the rebased branch keeps the existing JSON behavior for non-Nova models while fixing the Nova-specific request-shaping bug that was sending structured JSON blocks instead of their text representation.

…ol results

Nova models (Nova Pro, Nova Lite, Nova Micro) hallucinate when tool results
contain JSON content blocks ({'json': {...}}). They produce incoherent responses,
fabricate information, role-play both sides of conversations, and hallucinate
fake XML tool calls. Claude models handle JSON blocks correctly.

This fix adds model-aware JSON-to-text conversion in _format_request_message_content()
that automatically serializes JSON content blocks to their text representation
(json.dumps) when the model is an Amazon Nova variant. Non-Nova models continue
to receive JSON blocks as-is.

The detection follows the existing pattern used for tool result status handling
(_MODELS_INCLUDE_STATUS / _should_include_tool_result_status), adding a parallel
_MODELS_CONVERT_JSON_TO_TEXT list and _should_convert_json_to_text() method.

Closes strands-agents#1095
@giulio-leone
Copy link
Contributor Author

Refreshed this branch onto current main and revalidated it on the rebased head.

Repo-native gate (run twice with no code changes between passes):

  • uv run hatch run test-format
  • uv run hatch run test-lint
  • uv run hatch run test -- tests/strands/models/test_bedrock.py -q

Results on refreshed head 2dc7e3f847d79c7482cbc344e34485da41175f09:

  • ruff check: pass
  • mypy ./src: pass (Success: no issues found in 140 source files)
  • tests/strands/models/test_bedrock.py: pass (128 passed)

Direct branch-vs-main proof on exact source trees used the same temporary test file in both checkouts and asserted three things:

  1. Nova models serialize JSON-only tool results to text.
  2. Nova models serialize mixed text+JSON tool results so the JSON portion becomes text.
  3. Non-Nova models still preserve raw JSON tool-result blocks.

Proof result:

  • Refreshed branch: 3 passed
  • Current main (fd8168a53... at verification time): 2 failed, 1 passed

The current main failures are the exact regression this PR fixes:

  • Nova JSON-only tool results still come through as raw {"json": ...} blocks instead of {"text": json.dumps(...)}.
  • Mixed Nova tool results still leave the JSON block untouched instead of serializing it to text.
  • The non-Nova control still passes on both trees, so the fix remains scoped to Nova behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Amazon Nova Models Hallucinate with JSON Content Blocks in Tool Results

1 participant