feat(evaluation): add DatasetClient and dataset management service provider#490
Closed
jariy17 wants to merge 18 commits into
Closed
feat(evaluation): add DatasetClient and dataset management service provider#490jariy17 wants to merge 18 commits into
jariy17 wants to merge 18 commits into
Conversation
Add Dataset Management SDK support with: - DatasetClient: pass-through client for all 11 dataset APIs (create/get/list/update/delete datasets, versions, examples) with 6 _and_wait helpers for async operations - ServiceDatasetProvider: fetches datasets from the service and returns SDK Dataset objects compatible with OnDemandEvaluationDatasetRunner and BatchEvaluationRunner - Unit tests (20 tests) and integration tests (17 tests, verified against live AWS)
Switch ServiceDatasetProvider from list_dataset_examples pagination to downloading the JSONL file via the presigned downloadUrl from GetDataset. Single HTTP request is simpler and faster for large datasets.
- helpers.py: get_or_create_agent_runtime(), make_agent_invoker() with retry logic and warmup for cold start handling - test_runners_with_service_dataset.py: OnDemandRunner + ServiceDatasetProvider end-to-end test (skipped until a working deployed agent is available — current account has 30s init timeout that prevents cold starts)
Update test_runners_with_service_dataset.py to use env-var config: - INTEG_AGENT_RUNTIME_ARN: skips if not set - BEDROCK_TEST_REGION: region matching the agent - Verified end-to-end: ServiceDatasetProvider → OnDemandRunner → real agent invocation → COMPLETED
- ServiceDatasetProvider: accept client in __init__ (eliminates region_name) - ServiceDatasetProvider: validate schemaType against supported runner schemas - ServiceDatasetProvider: proper error message on download failure - Remove helpers.py (not needed) - Add unit tests for unsupported schema and download failure cases
- ServiceDatasetProvider: import DatasetClient at top, default in __init__ - ScenarioExecutor: add schema_type field, override in Predefined/Simulated - ServiceDatasetProvider: collect supported schemas dynamically from executors - delete_dataset_and_wait: add DELETE_FAILED as failed status
…dule level - Add schema_type field to Scenario base, PredefinedScenario, SimulatedScenario - Remove schema_type from ScenarioExecutor (doesn't belong there) - Move _parse_scenario from FileDatasetProvider to module-level function - SUPPORTED_SCHEMA_TYPES derived from Scenario classes directly
- Add timeout=60 to requests.get() for presigned URL download (#1) - Use r.content.decode("utf-8") instead of r.text for explicit encoding (#3) - Replace model_fields introspection with plain constant set (#8) - Guard __getattr__ against recursion when _cp_client not initialized (#5) - Remove dead _mock_client function from tests (aws#11)
…entServiceProvider Addresses PR review feedback: the name makes explicit which service the provider loads datasets from (Dataset Management Service).
Dispatch on file extension: paths ending in .jsonl are read line-by-line
(one scenario per line). All other paths keep the existing
{"scenarios": [...]} JSON shape.
Adds 8 unit tests covering predefined/simulated/mixed JSONL content,
blank-line tolerance, malformed lines, and extension dispatch.
…d_wait A version-specific delete (datasetVersion provided) does not remove the dataset itself — it transitions the dataset to UPDATING and back to ACTIVE. The previous waiter polled for ResourceNotFoundException and timed out. Branch on whether datasetVersion is passed: - Without datasetVersion: poll until ResourceNotFoundException (DELETE_FAILED) - With datasetVersion: poll until ACTIVE (UPDATE_FAILED), return dataset dict Add unit tests for both version-delete paths (success + UPDATE_FAILED) and an integ test that creates two versions, deletes the oldest via delete_dataset_and_wait, and verifies the dataset stays ACTIVE.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #490 +/- ##
=======================================
Coverage ? 89.49%
=======================================
Files ? 84
Lines ? 7732
Branches ? 1157
=======================================
Hits ? 6920
Misses ? 515
Partials ? 297
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
Author
|
Re-opening as an in-repo PR to fix the fork-PR token permission issue (breaking-change check failed only on the comment-post step due to GitHub's read-only token policy for fork PRs). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DatasetClient, a high-level wrapper for Bedrock AgentCore dataset management operations (create/get/list/delete datasets and versions, upload/download via presigned URLs, polling helpers).DatasetManagementServiceProvider(formerlyServiceDatasetProvider) that loads evaluation scenarios from a managed dataset, alongside the existingFileDatasetProviderwhich now also supports JSONL files.Scenarioclasses so each owns itsschema_type, and moves_parse_scenarioto module level for reuse between file- and service-backed providers.BEDROCK_TEST_REGION), and consolidates test fixtures.Test plan
uv run pytest tests/bedrock_agentcore/evaluationpassesuv run pytest tests_integ/evaluationpasses against a configured AWS accountruff, line-length)DatasetManagementServiceProviderend-to-end