fix(ai-red-teaming): multimodal SageMaker SigV4 targets work end-to-end (v1.7.1)#95
Merged
Merged
Conversation
…-to-end (v1.7.1)
Probing an Amazon SageMaker /invocations endpoint from the TUI silently
produced zero real trials. Three bugs in the generated multimodal-target
workflow, each found via live probing against a real Gemma-3 vision endpoint:
1. SigV4 signing imported `boto3`, which isn't installed in the workflow-runner
venv (only `botocore` is), so every trial died with
`ModuleNotFoundError: No module named 'boto3'`. Sign via `botocore.session`
directly — botocore is always present. Applies to both the image target and
the multimodal custom target.
2. Injection-image paths were baked into the workflow as relative paths, which
don't resolve from the workflow's execution CWD -> image never loaded
(`image=False`). Resolve media paths to absolute in `_expand_media_paths`.
3. Request-template placeholders were never substituted: the generated
`.replace()` used double-brace `{{prompt}}`/`{{image_b64}}` literals while the
template uses single braces, so an undecodable image reached the container
and it returned `424 Failed Dependency`. Emit single-brace replacements.
Also add `aws_sigv4` as a first-class `custom_auth_type` for multimodal targets
(with `custom_region`/`custom_service`), and print the real endpoint URL as the
target instead of the never-invoked backing model id.
Verified end-to-end: TUI -> SigV4-signed POST -> HTTP 200 real model response,
confirmed by CloudWatch Invocations on the endpoint.
SageMaker ASR/audio endpoints (e.g. a JumpStart Whisper container) take a raw audio file as the request body, not JSON — the existing JSON request_template path can't drive them. Add custom_request_format='audio_bytes' (with custom_audio_content_type, default audio/wav): the generated target posts the first audio part's raw bytes with SigV4 signing and extracts the transcript via custom_response_text_path. Verified end-to-end from the TUI against a live SageMaker Whisper endpoint: time_stretch-transformed audio -> raw-bytes SigV4 POST -> HTTP 200 transcript.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Probing an Amazon SageMaker
/invocationsendpoint from the TUI silently produced zero real trials. Root-caused by live probing against a real Gemma-3-4B vision endpoint and fixed three bugs in the generated multimodal-target workflow:ModuleNotFoundError: No module named 'boto3'— SigV4 signing importedboto3, absent in the workflow-runner venv (onlybotocoreis present). Now signs viabotocore.sessiondirectly (image target + multimodal custom target).image=False— injection-image paths were baked in as relative paths, unresolvable from the workflow's execution CWD, so the image never loaded._expand_media_pathsnow resolves to absolute.424 Failed Dependency— request-template placeholders were never substituted because the generated.replace()used double-brace{{prompt}}/{{image_b64}}literals against a single-brace template, sending an undecodable image to the container. Now emits single-brace replacements.Also:
aws_sigv4is now a first-classcustom_auth_typefor multimodal targets, withcustom_region/custom_service.Validation
Invocationson the endpoint confirmed the signed POSTs actually reached SageMaker.botocore.session, single-brace substitution,content=_content, and the endpoint URL asTARGET_LABEL.Test plan
/invocationsvision endpoint withcustom_auth_type=aws_sigv4,custom_region,custom_service, an OpenAI-stylemessagesrequest template, and an image transform — confirm a real model response (HTTP 200), not a[target error].