Skip to content

Add safe tool audit events #171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chubes4 merged 2 commits into from
May 15, 2026
Merged

Add safe tool audit events #171

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
abc2571
Add safe tool audit events
chubes4 May 15, 2026
1312359
Fix tool audit lint issues
chubes4 May 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions docs/runtime-and-tools.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ Representative message roles include user, assistant, tool-call, and tool-result
array(
'messages' => $messages,
'tool_execution_results' => $tool_results,
'tool_audit_events' => $tool_audit_events,
'events' => $events,
'turn_count' => $turns_run,
'final_content' => $last_assistant_text,
Expand Down Expand Up @@ -161,6 +162,41 @@ Failure modes are normalized rather than thrown to the loop:
- executor exceptions are caught and returned as tool errors;
- executor arrays without `success` are wrapped as successful results.

## Tool Audit Events

When the conversation loop mediates tool calls, the result includes
`tool_audit_events` alongside the backwards-compatible `tool_execution_results`.
The audit events are the safe replay surface for generic observers: they include
stable hashes and normalized status, but do not include raw tool parameters.

Representative event shape:

```php
array(
'schema_version' => 1,
'type' => 'tool_call',
'turn_count' => 1,
'tool_name' => 'client/search_docs',
'tool_source' => 'client',
'parameters_sha256' => 'sha256:...',
'parameters_redacted' => true,
'success' => true,
'result_status' => 'success',
'result_sha256' => 'sha256:...',
)
```

Failed calls include `error_type` when the loop can classify the failure. The
core classifications are `tool_not_found`, `missing_required_parameters`, and
`executor_exception`.

Sensitive parameter keys such as `token`, `secret`, `password`, `authorization`,
`cookie`, `credential`, `nonce`, and `api_key` are redacted before hashing. Hosts
can customize deterministic redaction with the
`agents_api_tool_audit_parameters` filter. The legacy `tool_execution_results`
field still contains raw parameters for existing callers and should be treated as
caller-owned runtime data, not as the generic replay artifact surface.

## Visibility and action policy

The tool policy layer resolves which tools are visible and how each tool may execute. Public policy classes include:
Expand Down
186 changes: 184 additions & 2 deletions src/Runtime/class-wp-agent-conversation-loop.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ public static function run( array $messages, callable $turn_runner, array $optio
$messages = WP_Agent_Message::normalize_many( $messages );
$events = array();
$tool_results = array();
$tool_audit_events = array();
$conversation_complete = false;
$exceeded_budget = null;

Expand Down Expand Up @@ -198,6 +199,7 @@ public static function run( array $messages, callable $turn_runner, array $optio

$messages = $mediation_result['messages'];
$tool_results = array_merge( $tool_results, $mediation_result['tool_execution_results'] );
$tool_audit_events = array_merge( $tool_audit_events, $mediation_result['tool_audit_events'] );
$events = array_merge( $events, $mediation_result['events'] );
$conversation_complete = $mediation_result['conversation_complete'];
$exceeded_budget = $mediation_result['exceeded_budget'];
Expand All @@ -206,7 +208,10 @@ public static function run( array $messages, callable $turn_runner, array $optio
$result = WP_Agent_Conversation_Result::normalize( $result );
$messages = $result['messages'];
$tool_results = array_merge( $tool_results, $result['tool_execution_results'] );
$events = array_merge( $events, self::normalize_events( $result['events'] ?? array() ) );
if ( isset( $result['tool_audit_events'] ) && is_array( $result['tool_audit_events'] ) ) {
$tool_audit_events = array_merge( $tool_audit_events, $result['tool_audit_events'] );
}
$events = array_merge( $events, self::normalize_events( $result['events'] ?? array() ) );
if ( isset( $result['request_metadata'] ) && is_array( $result['request_metadata'] ) ) {
$last_request_metadata = $result['request_metadata'];
}
Expand Down Expand Up @@ -261,6 +266,7 @@ public static function run( array $messages, callable $turn_runner, array $optio
$final_result_data = array(
'messages' => $messages,
'tool_execution_results' => $tool_results,
'tool_audit_events' => $tool_audit_events,
'events' => $events,
'turn_count' => $turns_run,
'final_content' => self::extract_final_content( $messages ),
Expand Down Expand Up @@ -311,7 +317,7 @@ public static function run( array $messages, callable $turn_runner, array $optio
* @param int $turn Current turn number.
* @param callable|null $on_event Event sink.
* @param array<string, WP_Agent_Iteration_Budget> $budgets Named iteration budgets.
* @return array{messages: array, tool_execution_results: array, events: array, conversation_complete: bool, exceeded_budget: string|null}
* @return array{messages: array, tool_execution_results: array, tool_audit_events: array, events: array, conversation_complete: bool, exceeded_budget: string|null}
*/
private static function mediate_tool_calls(
array $result,
Expand All @@ -329,6 +335,7 @@ private static function mediate_tool_calls(
: array();
$tool_calls = $result['tool_calls'];
$tool_execution_results = array();
$tool_audit_events = array();
$events = array();
$complete = false;
$exceeded_budget = null;
Expand Down Expand Up @@ -385,6 +392,15 @@ private static function mediate_tool_calls(
'turn_count' => $turn,
);

$tool_audit_events[] = self::tool_audit_event(
$tool_name,
is_array( $parameters ) ? $parameters : array(),
$exec_result,
is_array( $tool_def ) ? $tool_def : null,
$turn_context,
$turn
);

// Add tool-result message to transcript.
$result_content = ( $exec_result['success'] ?? false )
? self::json_encode_safe( $exec_result['result'] ?? array() )
Expand Down Expand Up @@ -441,6 +457,7 @@ private static function mediate_tool_calls(
return array(
'messages' => $messages,
'tool_execution_results' => $tool_execution_results,
'tool_audit_events' => $tool_audit_events,
'events' => $events,
'conversation_complete' => $complete,
'exceeded_budget' => $exceeded_budget,
Expand Down Expand Up @@ -538,6 +555,171 @@ private static function emit_event( ?callable $on_event, string $event, array $p
}
}

/**
* Build a stable, safe audit entry for a mediated tool call.
*
* The legacy `tool_execution_results` field intentionally keeps raw
* parameters for existing callers. Audit events avoid raw parameter storage by
* default so transcripts can be used for replay attestation without leaking
* secrets into generic observers.
*
* @param string $tool_name Tool identifier.
* @param array $parameters Runtime tool-call parameters.
* @param array $result Normalized tool execution result.
* @param array|null $tool_definition Tool declaration, when available.
* @param array $context Turn context.
* @param int $turn Turn number.
* @return array<string, mixed> Audit event.
*/
private static function tool_audit_event( string $tool_name, array $parameters, array $result, ?array $tool_definition, array $context, int $turn ): array {
$safe_parameters = self::redact_tool_audit_parameters( $parameters, $tool_name, $tool_definition, $context );
$metadata = isset( $result['metadata'] ) && is_array( $result['metadata'] ) ? $result['metadata'] : array();
$error_type = isset( $metadata['error_type'] ) && is_string( $metadata['error_type'] ) ? $metadata['error_type'] : '';

$audit_event = array(
'schema_version' => 1,
'type' => 'tool_call',
'turn_count' => $turn,
'tool_name' => $tool_name,
'tool_source' => is_array( $tool_definition ) && is_string( $tool_definition['source'] ?? null ) ? $tool_definition['source'] : '',
'parameters_sha256' => self::stable_sha256( $safe_parameters ),
'parameters_redacted' => true,
'success' => (bool) ( $result['success'] ?? false ),
'result_status' => ! empty( $result['success'] ) ? 'success' : 'error',
'result_sha256' => self::stable_sha256( self::audit_result_summary( $result ) ),
);

if ( '' !== $error_type ) {
$audit_event['error_type'] = $error_type;
}

return array_filter(
$audit_event,
static fn( $value ): bool => '' !== $value
);
}

/**
* Redact tool parameters before hashing them for audit events.
*
* @param array $parameters Raw tool-call parameters.
* @param string $tool_name Tool identifier.
* @param array|null $tool_definition Tool declaration, when available.
* @param array $context Turn context.
* @return array Redacted parameters.
*/
private static function redact_tool_audit_parameters( array $parameters, string $tool_name, ?array $tool_definition, array $context ): array {
$redacted = self::redact_sensitive_values( $parameters );

if ( function_exists( 'apply_filters' ) ) {
try {
/**
* Filters parameters before Agents API hashes them into tool audit events.
*
* Callers can remove or normalize product-specific sensitive fields while
* keeping deterministic replay hashes. Returning a non-array falls back to
* the default redacted parameters.
*
* @param array $redacted Default redacted parameters.
* @param array $parameters Raw tool-call parameters.
* @param string $tool_name Tool identifier.
* @param array|null $tool_definition Tool declaration, when available.
* @param array $context Turn context.
*/
$redacted = apply_filters( 'agents_api_tool_audit_parameters', $redacted, $parameters, $tool_name, $tool_definition, $context );
} catch ( \Throwable $error ) {
// Audit redaction filters must not change loop results.
unset( $error );
}
}

return $redacted;
}

/**
* Redact obviously sensitive scalar fields in nested parameter arrays.
*
* @param mixed $value Value to redact.
* @param string $key Current key.
* @return mixed Redacted value.
*/
private static function redact_sensitive_values( $value, string $key = '' ) {
if ( is_array( $value ) ) {
$redacted = array();
foreach ( $value as $item_key => $item_value ) {
$redacted[ $item_key ] = self::redact_sensitive_values( $item_value, is_string( $item_key ) ? $item_key : '' );
}
return $redacted;
}

if ( '' !== $key && preg_match( '/(api[_-]?key|authorization|cookie|credential|nonce|password|secret|token)/i', $key ) ) {
return '[redacted]';
}

return $value;
}

/**
* Keep the audit result hash focused on normalized status, not raw payloads.
*
* @param array $result Normalized tool result.
* @return array<string, mixed> Hashable result summary.
*/
private static function audit_result_summary( array $result ): array {
$metadata = isset( $result['metadata'] ) && is_array( $result['metadata'] ) ? $result['metadata'] : array();

$summary = array(
'success' => (bool) ( $result['success'] ?? false ),
'tool_name' => is_string( $result['tool_name'] ?? null ) ? $result['tool_name'] : '',
'metadata' => $metadata,
);

if ( empty( $result['success'] ) ) {
$summary['error_sha256'] = self::stable_sha256( is_string( $result['error'] ?? null ) ? $result['error'] : 'Tool execution failed.' );
}

return $summary;
}

/**
* Hash data after recursively sorting array keys for deterministic output.
*
* @param mixed $data Data to hash.
* @return string sha256-prefixed hash.
*/
private static function stable_sha256( $data ): string {
$normalized = self::sort_for_hash( $data );
$encoded = self::json_encode_safe( $normalized );
if ( false === $encoded ) {
$encoded = '';
}

return 'sha256:' . hash( 'sha256', (string) $encoded );
}

/**
* Recursively sort associative arrays before hashing.
*
* @param mixed $value Value to normalize.
* @return mixed Normalized value.
*/
private static function sort_for_hash( $value ) {
if ( ! is_array( $value ) ) {
return $value;
}

$normalized = array();
foreach ( $value as $key => $item ) {
$normalized[ $key ] = self::sort_for_hash( $item );
}

if ( array() !== $normalized && array_keys( $normalized ) !== range( 0, count( $normalized ) - 1 ) ) {
ksort( $normalized );
}

return $normalized;
}

/**
* Resolve the tool executor from options.
*
Expand Down
38 changes: 38 additions & 0 deletions src/Runtime/class-wp-agent-conversation-result.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,18 @@ public static function normalize( array $result ): array {
$result['tool_execution_results'] = array();
}

if ( ! array_key_exists( 'tool_audit_events', $result ) ) {
$result['tool_audit_events'] = array();
}

if ( ! is_array( $result['tool_execution_results'] ) ) {
throw self::invalid( 'tool_execution_results', 'must be an array' );
}

if ( ! is_array( $result['tool_audit_events'] ) ) {
throw self::invalid( 'tool_audit_events', 'must be an array' );
}

foreach ( $result['tool_execution_results'] as $index => $tool_result ) {
$path = 'tool_execution_results[' . $index . ']';

Expand Down Expand Up @@ -88,6 +96,36 @@ public static function normalize( array $result ): array {
}
}

foreach ( $result['tool_audit_events'] as $index => $audit_event ) {
$path = 'tool_audit_events[' . $index . ']';

if ( ! is_array( $audit_event ) ) {
throw self::invalid( $path, 'must be an array' );
}

foreach ( array( 'type', 'tool_name', 'parameters_sha256', 'result_sha256' ) as $field ) {
if ( ! array_key_exists( $field, $audit_event ) || ! is_string( $audit_event[ $field ] ) || '' === $audit_event[ $field ] ) {
throw self::invalid( $path . '.' . $field, 'must be a non-empty string' );
}
}

if ( ! array_key_exists( 'schema_version', $audit_event ) || ! is_int( $audit_event['schema_version'] ) ) {
throw self::invalid( $path . '.schema_version', 'must be an integer' );
}

if ( ! array_key_exists( 'turn_count', $audit_event ) || ! is_int( $audit_event['turn_count'] ) ) {
throw self::invalid( $path . '.turn_count', 'must be an integer' );
}

if ( ! array_key_exists( 'success', $audit_event ) || ! is_bool( $audit_event['success'] ) ) {
throw self::invalid( $path . '.success', 'must be a boolean' );
}

if ( array_key_exists( 'error_type', $audit_event ) && ! is_string( $audit_event['error_type'] ) ) {
throw self::invalid( $path . '.error_type', 'must be a string when present' );
}
}

// Validate optional budget-exceeded status fields.
if ( array_key_exists( 'status', $result ) && ! is_string( $result['status'] ) ) {
throw self::invalid( 'status', 'must be a string when present' );
Expand Down
9 changes: 6 additions & 3 deletions src/Tools/class-wp-agent-tool-execution-core.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ public function prepareWP_Agent_Tool_Call( string $tool_name, array $tool_parame
if ( ! is_array( $tool_definition ) ) {
return array_merge(
array( 'ready' => false ),
WP_Agent_Tool_Result::error( $tool_name, "Tool '{$tool_name}' not found" )
WP_Agent_Tool_Result::error( $tool_name, "Tool '{$tool_name}' not found", array( 'error_type' => 'tool_not_found' ) )
);
}

Expand All @@ -42,7 +42,10 @@ public function prepareWP_Agent_Tool_Call( string $tool_name, array $tool_parame
WP_Agent_Tool_Result::error(
$tool_name,
sprintf( 'Tool "%s" requires the following parameters: %s.', $tool_name, implode( ', ', $validation['missing'] ) ),
array( 'missing_parameters' => $validation['missing'] )
array(
'error_type' => 'missing_required_parameters',
'missing_parameters' => $validation['missing'],
)
)
);
}
Expand Down Expand Up @@ -76,7 +79,7 @@ public function executePreparedTool( array $tool_call, array $tool_definition, W
try {
$result = $executor->executeWP_Agent_Tool_Call( $tool_call, $tool_definition, $context );
} catch ( \Throwable $throwable ) {
return WP_Agent_Tool_Result::error( $tool_call['tool_name'], $throwable->getMessage() );
return WP_Agent_Tool_Result::error( $tool_call['tool_name'], $throwable->getMessage(), array( 'error_type' => 'executor_exception' ) );
}

if ( ! array_key_exists( 'success', $result ) ) {
Expand Down
Loading
Loading