ViewImageMiddleware injected the viewed-image message from before_model and
removed it again from after_model. before_model, model, and after_model are
separate graph nodes, so every view_image turn cost two extra nodes and two
state writes, and up to 20MB of base64 sat in two checkpoints for the duration
of the model call. A run interrupted in that window (user cancel, restart)
stranded the payload in history for good.
Inject from wrap_model_call instead, so the message lives only in
ModelRequest.messages and is never returned as a state update:
- before_model/after_model (and the async pair) are replaced by
wrap_model_call/awrap_model_call; _remove_image_context_messages and its
RemoveMessage bookkeeping go with them. The async hook keeps the existing
asyncio.to_thread offload for the file read and base64 encode.
- _should_inject_image_message gates on request.messages rather than state, so
the decision is made against what the model will actually see.
- _inject sweeps this middleware's own message out of the request before
rebuilding it. Dropping after_model also drops the cleanup it did on every
call, so without the sweep a payload stranded by an older interrupted run
would ride along in every later request for the life of the thread. Matching
requires both the reserved id prefix and the server-owned marker, and Gateway
strips that marker from client input, so a user message is never dropped.
Chain position is unchanged, and wrap_model_call nests first-registered
outermost, so TokenBudgetMiddleware still sees the image message and enforces
the input budget against it.
Checkpoint rows that already hold a stranded payload keep it on disk. It is
inert -- never sent to a provider, and strip_data_url_image_blocks keeps it off
the wire -- and reclaiming it would mean keeping the node this change removes.
tests/test_view_image_middleware.py is rewritten around the new hook (43
tests): sync/async at unit and graph level, the stranded sweep, and the
client-message protection. Docs: middleware chain entry 23, Vision Support, the
middleware-execution-flow hook matrix and diagrams, and the
strip_data_url_image_blocks docstring.
cb24bc2699