feat(sdk): implement LargeFileSurgicalCondenser for optimized context managementFeature/surgical condenser

[vivekvjnk]

Summary

This PR introduces the LargeFileSurgicalCondenser, a specialized context management tool designed to mitigate context window bloat caused by large tool outputs (specifically images and large file reads from the file_editor tool).

Unlike standard windowing or summarization condensers, this "surgical" condenser follows a Post-Inference Cleanup strategy. It preserves raw, high-fidelity data (like base64 images) during the turn it is first viewed to ensure the agent has the necessary information for analysis. Crucially, it only replaces that data with a concise summary after the agent has responded, ensuring subsequent turns benefit from a lean context and increased KV cache efficiency without sacrificing the quality of the initial analysis.

Key Features

• Targeted Condensation: Specifically monitors observations from the file_editor tool (configurable).
• Post-Inference Cleanup: Only triggers condensation when an observation is followed by an agent action/response, ensuring the data has already been "processed."
• Smart Summaries: Attempts to correlate observations with their preceding actions to include file paths in the condensation summary (e.g., [Condensation] Viewed sample_image.png).
• Vision Optimization: Automatically triggers for any ImageContent, which is a primary driver of context bloat in multi-modal workflows.
• Configurable Thresholds: Includes a default byte-size threshold (10KB) for TextContent.

Changes

• Core: Added LargeFileSurgicalCondenser in openhands/sdk/context/condenser/.
• Tests: Comprehensive unit tests covering tool-name filtering, threshold logic, image detection, and sequence-based condensation triggers.
• Example: Added a new vision-agent example script in examples/06_custom_examples/vision_agent/ demonstrating the condenser in a multi-turn conversation.

Performance Impact

By surgically removing prefix-heavy data like base64 images after their first use, we significantly increase prefix-matching for KV caches in multi-turn interactions, reducing latency and token costs for long-running sessions.

Checklist

• If the PR is changing/adding functionality, are there tests to reflect this?
• If there is an example, have you run the example to make sure that it works?
• If there are instructions on how to run the code, have you followed the instructions and made sure that it works?
• If the feature is significant enough to require documentation, is there a PR open on the OpenHands/docs repository with the same branch name?
• Is the github CI passing?

[csmith49]

Unfortunately I don't think this works the way you might expect. If we see the pattern:

<prefix>
Action event: retrieve large file
Observation event: the large file
<suffix>

You probably want the resulting sequence to look like:

<prefix>
Action event: retrieve large file
Condensation: summary of large file
<suffix>

But LLM APIs expect every action event to have a matching observation event and will throw an exception if that isn't the case. We prevent these exceptions by filtering out unmatched actions and observations when constructing the View, so the actual resulting sequence is probably something like:

<prefix>
Condensation: summary of large file
<suffix>

[vivekvjnk]

Actually I was trying to implement something like this:

<prefix>
Action event: retrieve large file
Observation event: [Condensation] summary of large file
<suffix>

Here actual observation event is replaced with condensed observation with same event id.
Rationale here is to make minimal changes to the original event list while generating new view.
So it felt unnecessary to condense the preceding action event.
(I've updated the code to return observation event instead of condensation event)

[csmith49]

I like the idea! Reminds me of the condensers we had in v0 (like this one to limit the data produced by the old browser tool). There are two problems/concerns I'd like to see addressed before approval:

Condenser Robustness

As implemented, I believe this approach is modifying more of the event history than is expected. I left some comments on the changes highlighting my concerns.

My recommendation is to tweak the LargeFileSurgicalCondenser to use the CondenserBase interface, which will simplify things:

class LargeFileSurgicalCondenser(CondenserBase):
    def condense(
        self,
        view: View,
        agent_llm: LLM | None = None
    ) -> View | Condensation:
        # The view is a list of events we want to show the LLM
        # Instead of trying to replace existing events with a
        # Condensation, just do the surgery on the events
        # directly while constructing a new view

No need for condensation_requirement or get_condensation, you can just always return a modified View to get the behavior you want. Doing surgery directly on the events in the view will help prevent the action/observation mismatch I noted.

Looping

When we had a similar masking behavior for browser outputs, the agents would often get stuck in a loop. They'd find a page they want, follow a few links to build context, and by the time they were ready to act the first page had been masked. But attempting to reload that context would mask the second page, and reloading that would mask the third, and by the time everything was reloaded the first would be masked again.

We tried a lot of things to fix the looping, but nothing really did the trick (which is part of the reason why the browser output masking condenser wasn't ported from v0 to v1).

This implementation only keeps large files around for a single agent step, and that's probably fine for a single image. Have you noticed any looping with multiple images? Or large text files linked together?

[csmith49]

Minor nit: this repo uses Google style doc-strings, not reST

[all-hands-bot]

[Automatic Post]: It has been a while since there was any activity on this PR. @vivekvjnk, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

[vivekvjnk]

Hi All,
I'm working on resolving all the comments. Got stuck with some other work.
I will try to resolve all issues by tomorrow.

[all-hands-bot]

[Automatic Post]: It has been a while since there was any activity on this PR. @vivekvjnk, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

[vivekvjnk]

please review

[all-hands-bot]

[Automatic Post]: It has been a while since there was any activity on this PR. @vivekvjnk, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

[all-hands-bot]

[Automatic Post]: It has been a while since there was any activity on this PR. @vivekvjnk, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.