* fix(agents): bound viewed image context per model request
Keep recent image payloads within a count and byte budget, and move re-viewed paths to the newest position. Add focused regressions for budget enforcement and state ordering.
* test(agents): verify cumulative image budgets and re-view recovery
* fix(agents): clarify image omission and read-attempt limits
* docs(middleware): shorten image context guidance for CI budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(view-image): bound re-injected image context with per-request budgets
`viewed_images` only grows (#6825): every model call that follows a
`view_image` round re-encodes and re-sends every entry ever viewed, so a
long PDF-inspection thread pushes the request body past provider limits
(e.g. Anthropic's 32 MB) even though each single image is capped.
Admission now runs newest-first against two knobs on ViewImageMiddleware:
- `max_injected_images` (default 10)
- `max_injected_total_chars` (default 32M characters, so a single
max-size image stays injectable: 20 MiB source -> ~27 MB base64)
Over-budget entries keep their text bullet with a placeholder ("call
view_image again to re-view this image") and are never read from disk or
sandbox, bounding both the request size and the middleware's per-call
work. Existing threads with small image counts and sizes are unaffected.
Refs #6825
* fix(view-image): make re-viewing an omitted image actually rescue it
The placeholder told the model to "call view_image again to re-view"
an over-budget image, but the recovery advice could not work:
merge_viewed_images merged with `{**existing, **new}`, and updating an
existing key in Python preserves its original insertion position. The
admission pass walks `reversed(viewed_images.items())` purely by
position, so the re-viewed entry stayed at its old (oldest) slot, was
admitted last, and was omitted again — a wasted tool round.
merge_viewed_images now re-inserts updated keys at the end, so dict
order tracks recency and a re-viewed image is admitted first on the
next injection. Also fix the `_create_image_details_message` docstring,
which said entries are "admitted oldest-first" while admission runs
newest-first (content blocks are emitted oldest-first; the admission
decision is not).
Wiring the budget knobs into config.yaml stays a follow-up: both
production instantiation sites construct ViewImageMiddleware() with no
arguments today, and this PR keeps the surface minimal.
Tests: reducer unit tests for re-insertion order plus an end-to-end
regression that re-views an omitted image and asserts it is admitted
on the next injection (#6829 review).
* test: route four unguarded symlink fixtures through symlink_or_skip
`tests/support/symlinks.py` was added in #5414 so suites that need a real
symlink skip cleanly when the host cannot create one. On a stock Windows host
(no Developer Mode, not elevated) `Path.symlink_to` / `os.symlink` raise
`OSError: [WinError 1314]`, and four fixtures still called them directly, so
they reported hard errors instead of skipping:
- tests/test_shared_reset_marker.py::test_marker_path_follows_a_symlinked_config
- tests/test_view_image_middleware.py::test_host_image_copy_is_scoped_to_current_user_and_thread
- tests/test_review_skill_package_tool.py::test_review_skill_package_rejects_symlink_planted_in_callers_outputs
- tests/test_channel_file_attachments.py::TestInboundFileSandboxPerms::test_make_inbound_file_sandbox_readable_swap_after_lstat_does_not_follow_symlink
These were the only symlink-related failures in a Windows run of the full
backend suite; the other bare symlink call sites there are already guarded by a
`skipif` or a `try/except OSError`. The view-image test plants its symlink
after its scoping assertions, so those still run before the skip. The
inbound-attachment test stages a symlink from inside a monkeypatched `os.lstat`
to exercise a TOCTOU race; a host that cannot create a symlink cannot stage
that race at all, so skipping is the only honest outcome.
* test: make channel symlink race fixture portable
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Store validated image bytes in the configured content-addressed blob backend and resolve checkpointed refs before local compatibility paths.
Refs: #4189
Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
* fix(sandbox): scope viewed image host reads to the current thread
* docs(gateway): trim agent guidance to fit CI limit
* fix(images): honor thread data paths for saved host copies
* test(images): align authorization fixture with thread paths
---------
Co-authored-by: YxinMiracle <“939157765@qq.com”>
ViewImageMiddleware injected the viewed-image message from before_model and
removed it again from after_model. before_model, model, and after_model are
separate graph nodes, so every view_image turn cost two extra nodes and two
state writes, and up to 20MB of base64 sat in two checkpoints for the duration
of the model call. A run interrupted in that window (user cancel, restart)
stranded the payload in history for good.
Inject from wrap_model_call instead, so the message lives only in
ModelRequest.messages and is never returned as a state update:
- before_model/after_model (and the async pair) are replaced by
wrap_model_call/awrap_model_call; _remove_image_context_messages and its
RemoveMessage bookkeeping go with them. The async hook keeps the existing
asyncio.to_thread offload for the file read and base64 encode.
- _should_inject_image_message gates on request.messages rather than state, so
the decision is made against what the model will actually see.
- _inject sweeps this middleware's own message out of the request before
rebuilding it. Dropping after_model also drops the cleanup it did on every
call, so without the sweep a payload stranded by an older interrupted run
would ride along in every later request for the life of the thread. Matching
requires both the reserved id prefix and the server-owned marker, and Gateway
strips that marker from client input, so a user message is never dropped.
Chain position is unchanged, and wrap_model_call nests first-registered
outermost, so TokenBudgetMiddleware still sees the image message and enforces
the input budget against it.
Checkpoint rows that already hold a stranded payload keep it on disk. It is
inert -- never sent to a provider, and strip_data_url_image_blocks keeps it off
the wire -- and reclaiming it would mean keeping the node this change removes.
tests/test_view_image_middleware.py is rewritten around the new hook (43
tests): sync/async at unit and graph level, the stranded sweep, and the
client-message protection. Docs: middleware chain entry 23, Vision Support, the
middleware-execution-flow hook matrix and diagrams, and the
strip_data_url_image_blocks docstring.
* fix(agents): stop persisting base64 image data in checkpoint state (#4138)
The viewed_images state field stored full base64-encoded image data,
which was duplicated across every subsequent checkpoint (O(n * steps)
growth). A single 1MB image viewed early in a conversation would be
re-stored in every checkpoint for the rest of the session.
Changes:
- ViewedImageData: replace base64 field with lightweight metadata
(mime_type, size, actual_path)
- view_image_tool: store only metadata in state, no base64 encoding
- ViewImageMiddleware: read image files from disk on-demand in
before_model and encode base64 temporarily for the model call
- Update all tests to use the new metadata-only format
This is the first step of #4138. The base64 data is no longer in
persistent state, but the injected HumanMessage (with base64 content)
still appears in the checkpoint for the step where it was injected.
Checkpoint retention policies and large tool result dedup are separate
follow-up items.
* fix(agents): address review feedback on #4140
- view_image_tool: remove stale 'convert to base64' comment, replace with
'validate contents'; drop redundant image_size reassignment and add a
TOCTOU guard that rejects files changed between stat() and read().
- view_image_middleware: extract _read_image_as_data_url helper that
re-checks size against the recorded value AND the absolute cap
(_MAX_IMAGE_BYTES). Document the trust assumption for actual_path
(server-set, not client-settable) in the helper docstring.
- view_image_middleware: abefore_model now runs the blocking read+encode
via asyncio.to_thread to avoid stalling the event loop on up to 20MB
images.
- tests: add coverage for OSError during read, file-changed-since-view
(TOCTOU), and size-exceeds-cap branches.
* docs(spec): MiniMax integration for generation skills + new music skill
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(plan): MiniMax generation providers implementation plan
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(skills): add importlib loader + FakeResp for skill tests
* test(skills): register loaded module in sys.modules; raise requests.HTTPError in FakeResp
* feat(image-generation): add MiniMax provider with env auto-detect
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(image-generation): guard unknown provider, derive ref MIME, strengthen tests
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(video-generation): add MiniMax provider with async poll/download
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(video-generation): surface base_resp errors while polling; add timeout test
* feat(podcast-generation): add MiniMax t2a_v2 provider with env auto-detect
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(podcast-generation): restore TTS credential guard; add volcengine + voice tests
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(music-generation): new MiniMax music skill via skill-creator
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(music-generation): treat empty lyrics as absent; test no-audio-data path
* refactor(skills): add request timeouts to MiniMax network calls
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Potential fix for pull request finding 'Explicit returns mixed with implicit (fall through) returns'
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* fix(models): strip inconsistent user-message names for MiniMax chat
DeerFlow middlewares tag user messages with provenance names (user-input, summary, loop_warning); langchain serializes them into the OpenAI-compatible payload and MiniMax rejects mismatched user-message names with "user name must be consistent (2013)". PatchedChatMiniMax now drops the per-message name from user-role messages. Point the config.example MiniMax models at PatchedChatMiniMax so they also get reasoning_content mapping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(image-generation): MiniMax sends JSON prompt field, guard 1500-char limit
MiniMax image-01 takes one text string capped at 1500 chars, but the skill was sending the whole structured JSON. The MiniMax provider now extracts the JSON `prompt` field (relying on prompt_optimizer to expand it) and fails fast with a clear error before calling the API when that field exceeds 1500 chars. Authoring stays provider-agnostic; Gemini still receives the full JSON.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(podcast-generation): per-provider TTS concurrency and retry/backoff
Each TTS provider owns its concurrency internally — MiniMax runs single-threaded to reduce rate-limit failures, Volcengine keeps 4 workers — with automatic retry and backoff on transient HTTP and base_resp errors. No caller-facing concurrency knob.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(skills): address Copilot review comments on generation skills
- video: add raise_for_status + timeout to the Gemini download/POST/poll calls so non-2xx responses surface as clear HTTP errors instead of JSON/KeyError or hangs
- video: check the task Fail status before the generic base_resp check so the failure keeps its task_id context
- video/image: create the output file parent directory before writing (matching music-generation) so nested output paths do not raise FileNotFoundError
- music: require a non-empty prompt and fail fast with ValueError instead of sending an empty prompt to the API
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(scripts): reclaim dev ports across worktrees in make stop/dev
All deer-flow worktrees (main checkout + linked worktrees) hardcode the same dev ports (8001/3000/2026), so a service started from any worktree must be reclaimable from another. stop_all now resolves the set of worktree roots (DEERFLOW_ROOTS) and treats a process as deer-flow-owned when its open files live under any of them. It also force-kills survivors on 2026 alongside 8001/3000, fixing `make dev` aborting on the nginx port preflight when a prior nginx lingered on 2026.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(view-image): hide the injected image-context message from the UI
ViewImageMiddleware injects a HumanMessage (text + base64 images) so the vision model can see viewed images, but it was the only internal injector that set neither hide_from_ui nor a hidden name, so it leaked into the chat UI (and IM channels) as a user bubble reading "Here are the images you've viewed:". Mark it with additional_kwargs={"hide_from_ui": True}, matching todo/dynamic_context injections, which the frontend isHiddenFromUIMessage and the channel sender already honor. The model still receives the full content.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(minimax): mark M2.7 models as text-only (no vision)
MiniMax M2.7 / M2.7-highspeed do not support vision; only M3 does. The
provider config asserted vision support for M2.7 in four places.
- config.example.yaml: 4 M2.7 entries -> supports_vision: false
- backend/docs/CONFIGURATION.md: M2.7 + highspeed -> supports_vision: false
- wizard: add LLMProvider.model_vision_overrides + extra_config_for() so
selecting an M2.7 model writes supports_vision: false while M3 (default)
keeps vision; wire it through setup_wizard.py
- tests: M2.7-highspeed fixture -> supports_vision=False; add
test_minimax_vision_is_per_model
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* test: add unit tests for ViewImageMiddleware
- Add 33 test cases covering all 7 internal methods plus sync/async
before_model hooks
- Cover normal path, edge cases (missing keys, empty base64, stale
ToolMessages before assistant turn), and deduplication logic
- Related to Q2 Roadmap #1669
* test: add unit tests for ViewImageMiddleware
Add 35 test cases covering all internal methods, before_model hooks,
and edge cases (missing attrs, list-content dedup, stale ToolMessages).
Related to #1669
