* fix: rebuild todo reminders after context compaction
* chore: remove implementation plan from PR
* refactor: share todo reminder message name
* docs: keep agent guidance within CI size budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(middlewares): end length-capped turns cleanly, prevent todo re-engagement, annotate write_file budget
When a model hits its per-response output cap (finish_reason=length) while
emitting a write_file tool call, ModelLengthFinishReasonMiddleware suppresses
the truncated call and stamps model_length_termination. TodoMiddleware must
not re-engage (jump_to=model) on such a capped turn -- doing so re-emits the
same oversized call into the same cap, producing up to 3 futile responses
with junk fragments instead of a clean truncation notice.
Changes:
- TodoMiddleware.after_model: skip completion reminder jump when
additional_kwargs.model_length_termination is present (follows the existing
deerflow_error_fallback precedent).
- ModelLengthFinishReasonMiddleware: always append the length notice when
tool calls were suppressed, even when partial text survived (collapses the
visible-content ternary). Fixes a latent bug in append_visible_text that
silently dropped string content.
- tools.get_available_tools: annotate write_file's model-visible description
with the model's configured max_tokens output budget. Guarded extraction
safely handles missing or non-numeric tokens, and the tool is cloned via
model_copy to keep module-level singletons immutable across assemblies and
prevent guidance leakage to unbudgeted models.
- release_policy_parameters() updated for both middlewares.
- AGENTS.md chain entries (#20, #35) and module docstrings updated within
AG002 guidance limits.
- Tests: 8 new/focused unit tests + 1 updated pin + 1 real create_agent()
integration test reproducing the incident (thread b1723286).
* fix(tools): use effective model cap for write_file guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): report non-mapping Honcho backend_config values as ValueError
failure_policy, workspace_overrides and user_peer_overrides were read with a
falsy-only `or {}` fallback, so a truthy non-mapping (a bare string, a YAML
list) reached .get/.items and escaped as a bare AttributeError from inside
backend construction. Route all three through one _mapping helper that keeps
falsy values meaning "unset" and names the offending key as a ValueError, the
posture the mem0 and OpenViking backends already take.
* fix(memory): name the Honcho numeric knobs that cannot be cast
timeout_seconds / connect_timeout_seconds / message_char_limit /
max_injection_chars still reached float() / int() with a YAML null or a
mapping, so the operator got a TypeError naming neither the key nor the
config file. Narrow all four through one _number helper that keeps falsy
values meaning "unset" the way the sibling api_key / storage_path scalars
already do, and reports a value that cannot be cast as a ValueError on the
key. Numeric strings keep parsing, since that is what float/int accept.
* fix(memory): report non-numeric backend_config knobs by name in mem0 and OpenViking
mem0 casts top_k, score_threshold, max_injection_chars and timeout_seconds, and
OpenViking casts timeout_seconds, max_seen_message_ids, retrieval.top_k and
retrieval.max_injection_chars, straight through int()/float(). A knob written
without a value in YAML therefore escapes as "TypeError: int() argument must be
a string..." from inside backend construction, naming neither which knob nor
which file is wrong, and a non-numeric value escapes as the equally anonymous
"could not convert string to float".
Both now resolve numeric knobs through a helper that keeps a value-less key at
its default, the way the same dicts already treat failure_policy and
allow_insecure_http, and turns an uncastable value into a ValueError that names
the knob. OpenViking's score_threshold keeps None as a meaningful value rather
than a default to fall back to.
* feat(middleware): add deterministic PII redaction for model-bound context
* fix(middleware): claim national IDs before cards, redact Command results, preserve ToolMessage fields
- Reorder detectors so checksum-gated national IDs run before the credit-card
detector; an 18-digit resident ID whose digit run also passes Luhn is no
longer mislabeled [CREDIT_CARD_n] (review finding, reproduced at 0a2a9d0)
- Redact ToolMessages carried in Command.update.messages, mirroring
ToolResultSanitizationMiddleware's dc_replace pattern
- Rebuild redacted ToolMessages via model_copy so artifact and
response_metadata survive
- Extend the numbered middleware chain in agents/middlewares/AGENTS.md
* fix(middleware): span one redactor per Command result; refresh stale AGENTS.md entry range
- Placeholder numbering now continues across every ToolMessage carried in a
single Command result (one _Redactor per _redact_result call) instead of
restarting per message
- The renumbered AGENTS.md chain still referenced entries 9-12 in the
ToolReceiptMiddleware entry; it now reads entries 10-13
* docs(agents): trim PiiRedactionMiddleware entry to fit the AGENTS.md chain budget
The main merge (fb36e0e) pushed the effective middlewares chain to 98341
bytes, 37 over the 98304 hard limit checked by agent-guidance (AG002).
Compress the entry while keeping the load-bearing facts: config gate, both
interception points incl. Command coverage, detector order rationale,
per-result numbering continuity, irreversibility, memory follow-up.
* fix(middleware): redact compaction input and reinjected summaries; harden detectors
Review round 3 on #5527:
- [P1] SummarizationMiddleware invokes its summary model directly from
before_model, outside PiiRedactionMiddleware's wrap_model_call, so raw
thread state reached the summary model and reinjected summaries carried
raw PII into model-bound context. Add a shared redact_text() seam: the
compaction prompt is redacted in _build_summary_prompt (app_config
already flows into the middleware) and DurableContextMiddleware redacts
summary_text at reinjection via a new pii_redaction_config knob wired
at both assembly sites.
- [P2] CUIT is 2+8+1 digits, not 2+10+1.
- [P2] Digit-anchored patterns use digit-aware lookarounds instead of
Unicode \b, which CJK characters defeat (身份证110105… / 手机号138…).
- [P2] The international-phone pattern no longer treats newlines as
separators, so a candidate cannot swallow the following numeric field
and then fail validation as a whole.
* fix(pii): redact title input and reserve summary placeholders
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(utils): return empty text for content-less messages
message_content_to_text fell through to str(content), so a message whose
content is None yielded the truthy literal "None". Two call sites already
work around it with a local `or ""` and name the helper in the comment; the
subagent executor's `text if text else "No response generated"` fallback and
the archive's `if not text: continue` skip cannot work around it, so a
content-less terminal turn was reported as an answer of "None" and a
content-less LLM error fallback surfaced "None" instead of its error_detail.
* test(utils): cover non-None content compatibility and document fallbacks
* test(utils): cover contentless task history and refresh guard comments
---------
Co-authored-by: Lengshuang <90967079+Lesereingrape@users.noreply.github.com>
Co-authored-by: JasonH <4430962+yang0228@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): close delegations a stopped run left in progress
Every task call is recorded in the delegation ledger as in_progress and
only moves on when its ToolMessage arrives. When the user stops a run
while a subagent is running, the task tool re-raises the cancellation
and no ToolMessage is written, so the entry stayed in_progress for the
rest of the thread and every later model call was told "already
delegated; do NOT delegate again; wait for or build on the result".
When a run starts with a new user message, mark entries that an earlier
run left in_progress and that have no ToolMessage as cancelled. Resumed
runs, which have no new user message, keep the current behaviour.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(agents): share the run-opening boundary between capture and closure
Also pin that in_progress entries without a run_id are never closed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(agents): clarify legacy delegation reply handling
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(knowledge): add verifiable RAGFlow source citations
* docs(knowledge): scope RAGFlow guidance to its own directory
* fix(knowledge): preserve citations through rendering and budgets
* feat(authz): filter per-caller skill visibility on the listing surfaces (#4063 Phase 4)
GET /api/skills, GET /api/skills/custom, and GET /api/skills/{name} now
filter the user-scoped catalog through filter_resources(principal,
"skill", ...) — mirroring list_models. Anonymous callers are unfiltered;
provider errors follow authorization.fail_closed (fail-closed -> empty
listing / 404, fail-open -> full listing). An invisible skill on the
detail surface returns the standard 404 so the endpoint cannot become an
existence oracle the filtered list closed. Management endpoints stay
require_admin_user-gated; runtime activation is #4541's layer.
resolve_skill_authorization joins resolve_model_authorization as a thin
sibling over a shared _resolve_route_scoped_authorization core.
* docs(authz): reflect per-caller skill visibility in OpenAPI metadata and implementation notes (#5489)
Address the two non-blocking review findings on #5489:
- The three user-facing GET routes (/skills, /skills/custom,
/skills/{name}) now say in their /docs-visible descriptions that
authorization filters the response (hidden skills 404 on detail).
- Add the dated Phase 4 decision-log entry to the authorization
implementation notes, per the convention of every prior merged
authz PR: listing-visibility semantics, the 404-vs-403
existence-oracle rationale, anonymous-caller behavior, and the
#4541 rebase reconciliation points (config.example.yaml roles
comment + this file's decision log).
* docs(authz): move route guidance into Gateway module guide
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Layer 1 quantized read_file's line range into 200-line buckets, which
erased the offset inside a bucket: every read shorter than a bucket
collapsed onto its neighbours. Five sequential 40-line reads hashed
identically and tripped the hard stop, ending the run with a forced final
answer and stop_reason=loop_capped — on exactly the ranged reads that
read_file's own truncation notice tells the model to make.
Bucketing cannot separate progress from repetition in general: an equality
key can only approximate range overlap, and the approximation was erasing
the offset that distinguishes the two. Key on the exact window instead,
with an omitted end_line kept open-ended so a bare read and an explicit
start_line=1 still share one key.
Repeating a single range is still caught at the same threshold, and a read
loop that varies its bounds remains covered by the per-tool frequency
layer.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): keep the delegation ledger and summary in create_deerflow_agent graphs
The SDK factory chain had no DurableContextMiddleware. SubagentLimitMiddleware
counts a run's delegations from the ledger that middleware writes, so the
per-run subagent total never tripped, and DeerFlowSummarizationMiddleware
keeps compacted history in summary_text, which only that middleware puts back
into model requests, so a summarized factory graph lost its history.
Add it after ToolErrorHandlingMiddleware, ahead of summarization, where
make_lead_agent has it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(agents): make RuntimeFeatures(token_budget=True) enforce the budget
The factory built TokenBudgetMiddleware from TokenBudgetConfig(), whose
enabled flag defaults to False, and every hook returns early on it. A graph
created with token_budget=True had no warning, no hard stop and no
token_capped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(agents): coalesce system messages in create_deerflow_agent graphs
DurableContextMiddleware injects its authority contract as a second
SystemMessage. The lead and subagent chains pair it with
SystemMessageCoalescingMiddleware because strict backends reject that; the
factory now does the same.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: trim inherited harness guidance below chain limits
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash
Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md
(issue #5160, tracker #5129) in the slice order of the spec's §16.
Slices:
- A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY
admission pinning (both server-owned sets + worker hoist); latest-only
request-scoped <project> block via DynamicContextMiddleware
wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID
prefix + marker + provenance, never persisted); journal audit
fingerprints; Instructions tab.
- B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with
locked check-and-set; hash-qualified immutable shelf storage with
Paths helpers; upload/list/content/delete-to-trash routes; project
delete trashes the shelf in-transaction; request-scoped bounded
<documents> index with honest count/shown + actionable overflow note;
list_project_documents/read_project_document tools registered only on
pinned runs; PAT allowlist + drift guards; blocking-IO anchors.
- C: shared thread-upload ingestion service (uploads router refactored to
parity); POST from-thread with provenance; attach-to-thread with
lock-staged copy (archived source allowed); read-only thread-files
view with per-group truncation reporting.
- D: restore (restored/merged/not_found/no_target/content_missing; no
file moves), purge (continuous row lock across unlink/delete/commit,
retryable on FS errors), retention sweep (lazy + startup, 24h orphan
guard, row-side reconciliation never deletes).
- E: Documents tab (shelf + conversation-files browser, provenance,
archived banner, content-missing rows), /workspace/trash route,
sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks
+ specs.
Review hardening folded in (10 rounds, all with tests):
- force active shelf content (HTML/XML family) to download; nosniff on
artifact + content responses; unified unsandboxed-iframe PDF preview
(fixes the pre-existing Chromium sandbox blank in the artifact viewer)
- scope document trash to the URL project under the document lock
- atomic no-overwrite filename reservation for ALL ingestion (seeded
claims + os.link commit with suffix retry; same-name re-upload now
unique-names instead of replacing); hidden staging only, no visible
placeholders; lease cleanup on setup failure
- serialize conversion under the document lock with post-lock active
revalidation; drain locked filesystem work on cancellation; preserve
bytes when an insert's commit state is uncertain (including trashed
rows)
- original-integrity checks before serving text or cached conversions;
content_missing surfaced in list responses (UI reads the flag, no
409-probe); downloads always serve original bytes
- bounded streaming document reads with cached char counts; shelf limits
declared in middleware release identity
- thread-root confinement for from-thread sources; config fallback
rejects fractional/infinite values; composer counts staged
attachments; pending attachments persist until submission or removal;
in-flight instruction/rename edits survive save refetches; shelf and
trash pagination; conversation-file and thread-files pages stay
subscribed to refetches
Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md
contracts, config.example.yaml projects block.
Review follow-ups (head b4807477 → this revision):
- The trash retention sweep is split so repeated lazy triggers stay
bounded: the indexed expiry purge still runs on every trigger
(GET /api/trash/documents, POST /api/trash/purge) while the
O(all rows + all files) reconciliation is throttled to one run per
user per 15 minutes (process-local, per-user window). The startup
sweep now runs as a background task instead of blocking gateway
readiness, and shutdown awaits it (bounded).
- The export scrub (stripInternalMarkers) is fence- and indentation-aware
like the render path, so a pasted, fenced <project>/<documents> snippet
survives markdown export while real injected blocks (never fenced) are
still removed. Fence regexes moved to a dependency-free leaf module to
avoid the messages↔streamdown import cycle.
- The artifact viewer's PDF iframe no longer carries an added title
attribute (the upstream e2e contract locates it via :not([title])), and
the upstream artifact-preview spec now pins the new contract: PDFs
render unsandboxed, images keep sandbox="".
* fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title
- Shutdown cancelled only the shield around the background startup sweep,
so an all-users reconciliation that outlived the 5s budget kept walking
rows and files while the document repo and DB engine were disposed
underneath it. The wait now lives in `_shutdown_startup_trash_sweep`,
which cancels the task and drains it before worker exit: the shield
keeps the wait bounded, the cancel makes it final (CancelledError lands
at the sweep's next await, and `_run_startup_trash_sweep` only catches
`Exception`, so nothing swallows it).
- The browser-preview iframe lost `title={getFileName(filepath)}` in the
previous fix round, leaving the PDF frame without an accessible name
while its siblings keep theirs. Restore it (WCAG frame titles), assert
it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]`
instead of `iframe:not([title])`.
* fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel
`Task.cancel()` returns False when the sweep already finished inside the
window between the deadline firing and the cancel, so the shutdown log
claimed a cancellation that never happened. Branch on that outcome: the
warning stays for a real cancel, a late finish is logged at info, and both
paths still reap the task before worker exit.
* fix(projects): round-4 review — make Empty trash delete what it confirms
`POST /api/trash/purge` only ran the retention sweep, and the sweep's
candidate selection is age-gated, so a freshly trashed document survived
"Empty trash" even though the confirmation promises that every listed
document is permanently deleted. With one trashed row the route answered
`{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps
expired rows before listing, so the visible rows were normally ineligible
for the action by construction.
Empty trash now drives `purge_all_trashed`: the caller's trashed rows
(`list_all_trashed`, no age filter) each go through the same guarded,
row-locked `purge` as the single-document delete — bytes first, then the
row, in one transaction — so a row restored mid-flight is skipped instead of
force-deleted, and an unlink failure rolls that row back and answers 500 with
a retryable message. Retention expiry stays where it was: the sweep's
`purge_candidates` is now the only age-gated selection, and the lazy
retention sweep still runs on the listing and at startup.
Tests: the router suite replaces the retention-gated expectation with the
reviewer's repro (fresh row purged, bytes unlinked, shelf and other users'
trash untouched, a failing unlink stays retryable and 500); a blocking-I/O
anchor drives the new entry point through the offload; the mocked e2e covers
the action end to end; a new real-backend spec performs it against the real
gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE
and the phase-2 design docs (en+zh) state the age-independent contract.
* fix(agents): keep queued guard warnings when a model call is retried
LoopDetectionMiddleware, TokenBudgetMiddleware and ToolProgressMiddleware
pop their queued warning/hint before calling the model. When the call
raises, LLMErrorHandlingMiddleware (outside them) retries by running
their wrap_model_call again, and by then the queue is empty, so the
retried request goes out without the warning. Loop detection and the
token budget have already marked it as sent, so it is never queued
again, and a loop runs on to the hard stop unwarned.
Put the drained items back in front of the queue when the handler
raises, then re-raise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(agents): trim restored loop warnings from the tail and drop a dead helper
_restore_pending_warnings put the restored warnings at the front and then
trimmed the front, so if the cap ever fired it would drop exactly what it
restored. Trim the tail, as tool progress does. _augment_request had no
callers after the wrap_model_call change. Add the sync twin of the tool
progress retry test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(agents): drop tool progress's unused _augment_request
Its only remaining reference was a test name; the dedup that test checks lives
in _inject_hints.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(agents): remove provider tool-call blocks when guards strip calls
Token-budget and loop-detection hard stops, subagent-limit truncation,
and safety-finish-reason suppression removed calls from tool_calls and
the raw additional_kwargs payload, but left the provider's own
tool-call blocks in AIMessage.content. Provider adapters re-serialize
those blocks: langchain_anthropic sends a tool_use block whose id is
not in tool_calls, and the OpenAI Responses input builder sends every
function_call block. ChatAnthropic stores any tool-calling response as
a block list, so a guard firing on a Claude tool call always left a
tool_use without a tool_result. A truncated subagent call failed the
next model request of the same run; a hard stop was checkpointed under
the same message id and failed every later turn of the thread.
clone_ai_message_with_tool_calls now trims content tool-call blocks to
the calls that remain on the message: tool_use and LangChain v1
tool_call/tool_call_chunk by id, Responses function_call and
custom_tool_call by call_id (their id is the fc_ item id), Google GenAI
function_call by id, and id-less blocks by name in order. Blocks for
calls still on invalid_tool_calls stay, because
DanglingToolCallMiddleware answers those calls with placeholder
results. The token-budget and loop-detection hard stops now build their
messages through the helper instead of their own copies, and
ClarificationMiddleware drops its private filter, which matched
Responses blocks by item id.
* docs(changelog): reference #5447 in the orphaned tool-call block entry
* fix(agents): skip id-matched calls in the id-less block budget
The name budget for id-less content tool-call blocks counted every
retained call, including calls whose own id-bearing block had already
matched. In mixed-shape content, a retained call "a" with a
function_call block carrying id "a" also let a same-named id-less block
survive, leaving the unpaired block this helper exists to remove.
Collect the retained ids that id-bearing blocks matched first, and build
the name budget only from retained calls outside that set. Content with
no id-bearing blocks keeps the full budget, so the Gemini path is
unchanged.
* fix(subagents): keep the subagent system prompt through context compaction
A subagent carries its whole system prompt (role, report contract, skills,
deferred tools) as the leading SystemMessage in state, because its agent is
built with system_prompt=None. Summarization only rescued tagged reminders and
the latest user message, so the first compaction summarized the prompt away
and every later model call in that subagent ran without its instructions.
Rescue system messages as well. The lead agent's prompt is not in state, and
its in-state SystemMessages are the tagged reminders already rescued, so the
lead chain is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(compaction): clarify preservation contract and trim agent guidance
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): keep token budget signals for runs without a run_id
#5410 moved every invocation without a non-empty string run_id onto
str(id(runtime)). Two things break on that key:
- SubagentExecutor passes the parent's run_id, None when the parent run has
none (LangGraph Server, direct create_deerflow_agent callers), and reads the
stop reason back with that None. The hard stop stored it under the id string,
so a token-capped subagent reported a clean completion to the lead.
- LangGraph gives each graph node its own Runtime wrapper, so the key changed
between after_model and the next model call: the budget warning was never
delivered, and each after_model counted every AIMessage in the thread.
Key those invocations by Runtime.control, as LoopDetectionMiddleware does,
release it in after_agent, and store the stop reason under the context run_id
as given.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(agents): keep an active invocation's budget key when the anchor map is full
The fallback anchor map was FIFO, so with 1000 run_id-less invocations on a
shared instance an active one could lose its anchor mid-run and restart with a
fresh budget. Move the anchor to the end on every lookup, as loop detection
does, and note why execution_info.run_id is not consulted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
McpRoutingMiddleware picked the latest user message with
is_real_user_message, which rejects every hide_from_ui message. A Human
Input Card reply is hidden but is still the user's current request, so
when the routing keyword lived only in the clarification answer the
deferred MCP tool was never auto-promoted and the model had to call
tool_search by hand.
Switch _latest_user_message to is_genuine_user_message, the same
predicate summarization_middleware already uses for this reason (#5416).
Fixes#5425
Summarization rescues the latest user message by id so the current
request survives compaction (#4882). It picked that message with
is_real_user_message, which rejects every hide_from_ui HumanMessage,
including Human Input Card replies. When a run starts from a card
reply, the older visible request was kept and the user's answer was
summarized away.
Use is_genuine_user_message, which accepts hidden messages that carry a
valid human_input_response and still skips other hidden injections.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(agents): keep the token budget across goal continuations of a run
TokenBudgetMiddleware cleared its usage in after_agent, and before_agent
marks every existing message as seen. A Gateway run re-enters the graph
for hidden goal continuations under the same run_id, so each
continuation started from zero and could spend another full budget,
even after the user turn had already hit the hard stop.
Keep the run's usage and warning state across graph entries, as
LoopDetectionMiddleware does since #5344. Only the per-message seen map
is dropped, and before_agent rebuilds it. Invocations without a run_id
in the context still clear everything.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(agents): normalize token budget run identity and test cleanup
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): reject forged framework-injection markers in run input
`is_genuine_user_message` treats `hide_from_ui` and a human `name="summary"`
as proof the framework authored a message, and `InputSanitizationMiddleware`
skips those — escaping a real reminder's blocks would corrupt trusted context.
Neither marker was server-owned, so an external caller could set either one and
place a raw `<system-reminder>` outside the user-input boundary markers, which
the lead-agent system prompt declares trusted internal framework data. The
`hide_from_ui` variant is also filtered out of the thread UI, so the forgery
was invisible where it landed.
Both markers are now stripped from untrusted input, on the run path and on the
thread-state mutation route that writes straight into a checkpoint. Framework
injection happens inside the graph and never crosses this boundary, so nothing
the framework does is affected, and `trusted_internal` callers (IM channels,
the MCP task-notification launcher) keep writing hidden messages.
HumanInputCard replies are the one legitimate external `hide_from_ui`: the
frontend sends it alongside a `human_input_response` payload, so a message
carrying a valid one keeps the marker. That buys no bypass — the predicate
already classifies those as genuine, so they stay sanitized.
The name check reuses the predicate's own `_SUMMARY_MESSAGE_NAME` rather than a
fourth copy of the literal, and matches by `isinstance` exactly as the predicate
does: `HumanMessageChunk` is a `HumanMessage` whose `type` is not `"human"`, so
a type-based check would leave that subclass's marker settable. `name` is only
reserved on human messages — on a ToolMessage it is the tool's own name.
Three tests in test_gateway_services.py and test_message_provenance.py asserted
that a caller-supplied `hide_from_ui` survives. That assumption was the bypass;
they now pin the opposite, with a genuinely caller-owned key kept alongside to
prove the stripper is surgical.
* fix(agents): sanitize every genuine user message, not only the newest
The input guardrail scanned backwards for the first genuine user message and
returned, so only the newest turn was ever sanitized. The transformation is
request-scoped (`wrap_model_call`, never written to state), so thread state
keeps the raw text: once a newer turn arrived, the previous turn's payload was
replayed to the model verbatim, outside the boundary markers the lead-agent
prompt declares trusted framework data. The guardrail therefore held for
exactly one model call.
Reaching it needed no forged metadata and no crafted request body — type the
payload in one turn, then send anything at all in the next. A single request
carrying two user messages did it in one shot, since every message but the last
was skipped.
`_process_request` now walks the whole list and `_sanitize_message` owns the
per-message work; every existing branch (the `original_user_content` split for
upload turns, the multimodal rfind fallback, the metadata repair) is unchanged.
Framework-injected messages stay excluded by `is_genuine_user_message`.
Unexpected errors now fail open per message rather than per request. Iterating
history widened the old blast radius: one unprocessable row would have dropped
sanitization for the whole request, handing an attacker the newest turn by
crafting an older one. `GraphBubbleUp` still propagates.
Side effect worth noting: each turn's rendering is now stable across model
calls. Previously a turn was wrapped on its own call and unwrapped on the next,
changing the prompt prefix behind the newest turn and defeating prompt caching.
test_only_processes_last_user_message pinned the old scope; it now pins that
every turn is processed, and keeps driving the `wrap_model_call` entry point.
* docs: record the message-metadata trust boundary and sanitization scope
`agents/middlewares/AGENTS.md` owns the depth for InputSanitizationMiddleware
and documented only the `original_user_content` half of its trust boundary. Left
alone it would teach an agent that `hide_from_ui` is caller-owned and that the
guardrail covers one turn — and the usual failure mode is an agent "restoring"
the behaviour it believes was lost. The entry now carries both markers, the
HumanInputCard exception, the whole-history scope, and the per-message fail-open
rule.
The note lives only there. The root and `backend/AGENTS.md` layers are
orientation that points at the module guides owning the depth, and
`backend/AGENTS.md` is inherited by every backend chain — prose added there
inflates more than twenty of them, and `scripts/check_agent_guidance.py` shows
the middlewares chain has about a kilobyte of room against its hard limit.
CHANGELOG.md and CHANGELOG_zh.md record it under Security, continuing the
existing prompt-injection lineage.
* fix(gateway): mark caller-hidden messages instead of stripping the marker
Review follow-up. Stripping a caller-supplied `hide_from_ui` closed the bypass
but broke three frontend senders that use the marker purely to keep a context
message out of the transcript: the quoted conversation context
(`buildHiddenConversationQuoteMessage`), the sidecar context prompt
(`buildHiddenSidecarContextMessage`), and the agent save command. None carries a
`human_input_response`, so the HumanInputCard carve-out did not cover them, and
nothing else hides them — `_is_branch_visible_message` and the frontend's
`isHiddenFromUIMessage` both key solely on `hide_from_ui`, and no backend reads
`conversation_quote_context` or `sidecar_context`. All three would have rendered
as user-visible chat bubbles.
The marker plays two roles and only one of them is a vulnerability. The security
requirement is that a caller-supplied marker cannot skip sanitization, not that
it cannot hide a message. So the roles are separated instead of the marker being
removed: the Gateway keeps it and stamps the server-owned `UNTRUSTED_INPUT_KEY`,
and the guardrail now asks `requires_input_sanitization` — the mark, else the
genuine-user test. Hidden stays hidden; untrusted content is sanitized either
way. The reserved `summary` name is handled the same way and no longer rewritten.
Marking rather than removing is also the safer shape in general: `hide_from_ui`
is read for presentation, journal persistence, memory filtering and IM outbound
as well, and this boundary should not silently change any of them.
`is_genuine_user_message` is deliberately left alone. `ToolReceiptMiddleware`
uses it for turn-boundary detection, where a caller's hidden context message must
keep counting as not user-authored; widening it there would move the ledger's
turn window. `requires_input_sanitization` sits beside it in `message_utils` so
the two questions can be compared.
The three tests that asserted a caller-supplied `hide_from_ui` is removed now
assert it survives and carries the mark — which restores the original intent of
the two provenance cases, whose comment already read "caller-owned keys must
survive".
* docs(gateway): rewrite normalize_input's docstring around the mark
Review follow-up. The paragraph still described the pre-4a3344f6 strip model and
contradicted both the implementation and the middlewares AGENTS.md paragraph
updated in that same commit: it called `hide_from_ui` server-owned, said
carrying it skips sanitization entirely, and repeated the premise this branch
disproved — that HumanInputCard replies are the only legitimate external use.
It now describes what the code does: the markers stay caller-owned and are
preserved because three frontend senders rely on `hide_from_ui` for hiding
alone, the message is stamped with `untrusted_input` instead, and
`requires_input_sanitization` sanitizes it anyway. `untrusted_input` joins the
server-owned inventory in the preceding paragraph, which is what makes the stamp
unforgeable and unclearable.
The three surrounding docstrings now also say that these functions mark as well
as strip; `_strip_external_message_metadata` had advertised only the removal,
leaving a reader no way to find the stamp from the name.
* fix(gateway): mark state writes whose message omits additional_kwargs
Review follow-up. The state-route half of the fix missed the most natural
request shape. `_strip_external_metadata_from_message_like` returned early when
`additional_kwargs` was absent or not a dict — there was nothing to strip — and
that early return also skipped the mark. A `POST /threads/{id}/state` body of
`{"values": {"messages": [{"role": "user", "name": "summary", "content":
"<system-reminder>…</system-reminder>"}]}}` therefore reached the checkpoint
unmarked. The messages reducer's `convert_to_messages` then supplies
`additional_kwargs={}`, so at model-call time `requires_input_sanitization` fell
back to `is_genuine_user_message`, which a `summary` name fails, and the forged
tag reached the model raw and outside the boundary markers.
A missing or non-dict `additional_kwargs` is now treated as empty for both the
strip and the mark. The identity return is kept for the case where nothing
changes, so an ordinary key-omitted message does not gain an empty dict just by
passing through. The run path was never affected: `normalize_input` coerces to
BaseMessage first, which always carries the dict.
Every existing state-write test supplied an `additional_kwargs` dict, which is
why this shape slipped through; the regression now covers it at the route and
end to end through the reducer into the guardrail.
While checking the neighbouring shapes, `_skips_input_guardrail` keyed off key
presence where `is_genuine_user_message` keys off truthiness, so
`hide_from_ui: False` — already covered without a mark — would have been
stamped. It now mirrors the predicate exactly, as its docstring claimed.
* docs(middlewares): compress the sanitization note to fit the guidance chain
main's growth left the middlewares AGENTS.md chain 84 bytes under its hard
limit, and the fuller wording did not fit. The load-bearing facts stay — the
markers are marked rather than stripped, and the scan covers every turn — since
those are the two an agent editing this middleware could otherwise get wrong.
The full model lives in the normalize_input, _mark_untrusted_framework_markers
and requires_input_sanitization docstrings.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(memory): add deterministic near-duplicate fact gate
Add an opt-in write-side gate for DeerMem (issue #5252): a proposed NEW
fact whose bounded token-Jaccard similarity to an existing fact in the
same user/agent scope AND category reaches
fact_dedup_similarity_threshold merges into that fact instead of being
appended — the existing id/content/createdAt are kept, confidence is
raised to the maximum, and the source is refreshed, with one
facts_merged_dedup metric increment recording the merge.
Defaults preserve the legacy behavior exactly; targeted updates by fact
id and the exact-content key check are untouched. Companion write-side
step to the read-side relevance/diversity work in #5251.
Refs #5252
Signed-off-by: pwd11 <fvdsrc@163.com>
* fix(memory): preserve corrections during fact deduplication
Signed-off-by: pwd11 <fvdsrc@163.com>
---------
Signed-off-by: pwd11 <fvdsrc@163.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(agents): elide superseded write_file payloads from model-bound requests
Step 2 of #5328. After a successful write_file the file on disk is the source
of truth, and the read-before-write gate forces a read_file before the next
modification of that path, so once a later successful read or write of the
same path exists the historical `content` argument is redundant with it. Long
report-writing runs (append-in-chunks) therefore carried every section twice,
once as the write argument and once as the following read output, until
summarization compacted the whole turn.
- ToolOutputBudgetMiddleware's model-call hooks now replace such superseded
content with a short deterministic placeholder pointing at read_file, in the
model-bound request only: state["messages"], checkpoints, receipts, loop
detection, and the run journal keep the original arguments, and nothing is
externalized to disk. The newest `keep_recent_writes` successful writes
(default 1) always stay visible; str_replace payloads are never touched; a
same-turn read never supersedes (parallel calls run in no fixed order); only
results stamped deerflow_tool_meta.status == "success" count, so failed,
gate-blocked, partial, or unstamped writes are never candidates.
- New `tool_output.elide_superseded_writes` (default on),
`tool_output.superseded_write_min_chars` (default 2000), and
`tool_output.keep_recent_writes` (default 1); config_version 41 -> 42 in
config.example.yaml and the Helm chart.
- The per-occurrence call/result pairing the gate introduced in #5329 moves
into the shared `tool_call_args.pair_tool_call_results` helper so both
policies pair the same way; the gate now uses it.
* fix(agents): scope tool-call result pairing to the issuing turn
Review finding on #5374 (P2): pair_tool_call_results consumed results from a
history-wide per-id queue, so an interrupted write_file with no result whose
tool-call id a later turn reused inherited that later call's success. With
the default elision the unconfirmed draft was then replaced by a placeholder
claiming the write succeeded, and the gate's blocked-call pairing had the
mirror-image hole.
Pair results the way DanglingToolCallMiddleware does: walk in document order,
open each AIMessage's calls, and let a ToolMessage answer only a still-open
call of the most recent preceding AIMessage. A result never answers a call
from an earlier turn, so the interrupted call stays unanswered (never a
candidate, never labeled blocked) and stray or duplicate results are ignored.
Regressions cover the helper, the superseded-write policy, and the gate.
* fix(agents): never rewrite tool-call ids duplicated within one AIMessage
Review finding on #5374 (P2): the policies select calls per occurrence, but
every provider surface is addressed by tool-call id, so when a malformed
provider payload repeats an id inside one assistant turn the rewriter could
only replace all of its occurrences at once. A failed write_file sibling then
took on the superseded successful call's path and elided content and was
presented as a success; the gate's blocked-call elision had the mirror-image
hole (a successful sibling rewritten into the blocked call).
rewrite_messages_tool_call_args now never offers an id that repeats within
its message to the selector and leaves those calls untouched on every
surface. Both policies are covered by the shared helper; regressions cover
the helper, the superseded-write policy, and the gate.
* fix(agents): skip unhashable tool-call ids in the duplicate-id guard
Review finding on #5374 (round 3): _duplicated_call_ids fed every id into a
Counter before the string guard, so a list or dict id from a malformed
provider payload raised TypeError out of wrap_model_call and failed the whole
model call whenever the history also held a rewrite candidate. The pre-PR
loop and pair_tool_call_results skip such ids; only this helper regressed.
Count non-empty string ids only, and pin it with regressions for the helper,
the superseded-write policy, and the gate.
* docs(agents): keep middleware guidance within size limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: add opt-in task notes and compacted history recall
* fix: validate task continuity state and preserve user answers
Honor explicit opt-out, preserve clarification replies and capture failure statuses, validate notebook writes, and clear branch archive references. Update the config version and audit optional LLM credentials, with regression and integration evidence.
* fix: align Helm config version with task continuity schema
* fix: preserve mixed task history and declare continuity policies
* fix: recover malformed history and evict archives atomically
Toggling a skill wrote resolved secrets into extensions_config.json. The
Gateway skill toggle and DeerFlowClient.update_skill loaded the file with
ExtensionsConfig.from_file(), which replaces every "$VAR" string with the
environment value (and an unset variable with ""), then serialized that
model back through to_file_dict(). A "$GITHUB_TOKEN" reference was
persisted as the plaintext token, and an unset reference was erased for
good. DeerFlowClient.update_mcp_config had the same flaw for every key
other than mcpServers.
Every writer now does a raw read-modify-write, the way the MCP router
already did: read_raw_extensions_config reads the on-disk JSON,
set_raw_skill_enabled changes only the target entry, and
validate_raw_extensions_config checks the candidate the way the runtime
will load it before the atomic write. When the file does not exist yet,
the Gateway seeds it with the cached skill states only, never the
resolved cached model. The MCP router's raw loader and candidate
validation delegate to the same helpers, so the rule lives in one place,
and to_file_dict() is removed so the unsafe serialization has no entry
point left.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): allow constructor kwargs on config-declared middlewares
extensions.middlewares entries may be a class-path string or {class, kwargs}.
String entries keep the zero-argument constructor. Unknown fields and blank
class paths fail at config validation. Constructor errors still fail at
agent creation.
Fixes#5311
* fix(extensions): coerce middleware kwargs to JSON types
YAML timestamps became datetime objects while JSON kept strings, so
constructors and to_file_dict() json.dump saw different types. Validate
kwargs as JSON types at config load, stringify dates, reject NaN and
other non-JSON values, and cover the raw-dict loader branch.
* style(extensions): wrap middleware Field description for ruff E501
make lint failed: the middlewares description was 289 chars (limit 240).
Wrap it and run ruff format on the two files this PR last touched.
* docs: compact configured middleware guidance to satisfy size limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(title): use filenames for attachment-only conversations
* docs: trim upload guidance to fit inherited size budget
* fix(title): bound attachment-only fallback titles
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(agents): elide blocked write payloads from model-bound requests
A write_file / str_replace call rejected by the read-before-write gate never
runs, yet its payload (up to 80 KB for a non-append write, unbounded for
append) stayed verbatim in every later model request: nothing in the chain
rewrites AIMessage tool-call arguments, and ToolOutputBudgetMiddleware only
budgets ToolMessage output. The gate demands a re-read plus a fresh call, so
the model re-emits the content anyway and the original is pure dead weight.
- ReadBeforeWriteMiddleware stamps `deerflow_write_block` on the blocked
ToolMessage and, in wrap_model_call, replaces the paired call's payload
fields (content / old_str / new_str) with a short deterministic placeholder
in the model-bound request only. state["messages"], receipts, loop
detection, and the run journal keep the original arguments; nothing is
externalized to disk, since a file reference to content the model must
re-derive after reading the target would only invite bypassing the gate.
- New `tool_call_args` helper rewrites every provider surface together
(structured tool_calls, raw additional_kwargs.tool_calls, tool_use content
blocks, tool_call_chunks) so strict providers never see them disagree; the
gate only supplies the policy (which calls, what placeholder).
- `read_before_write.elide_blocked_payloads` (default on) and
`read_before_write.elide_min_chars` (default 2000) configure it; the
runtime builder passes the config through and the middleware declares it
via release_policy_parameters.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs(agents): condense middleware guide entry 11 to fit the guidance budget
The agent-guidance CI check failed: the effective AGENTS.md chain for
agents/middlewares was 99673 bytes against a 98304-byte hard limit. The
chain already sat at 98459 on main, so the ReadBeforeWrite entry could not
grow. Rewrite entry 11 so it states the same facts (gate, lock scope,
fail-open, authorization scope, blocked-payload elision, shared
tool_call_args helper) in 1229 bytes instead of 2640; the chain is now
98262 bytes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(config): bump config_version for the read_before_write elision keys
Review follow-ups on #5329:
- `read_before_write.elide_blocked_payloads` / `elide_min_chars` are new
user-settable YAML keys, i.e. a config schema change, so bump
`config_version` 40 -> 41 in config.example.yaml; without it an existing
config.yaml gets no outdated-config warning and `make config-upgrade` has
nothing to signal.
- Say in the `elide_min_chars` description (and the example comment) that the
threshold and the placeholder's size figure are Python character counts,
not tokens: the same value spans roughly 3-4x in real context cost between
ASCII and CJK text.
- The builder wiring test now asserts only the wired `elide_min_chars` value
instead of the whole `ReadBeforeWriteConfig` dump, so future knobs do not
have to edit an unrelated test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* chore(helm): bump chart config_version to 41
validate-chart's config_version drift check failed after config.example.yaml
moved to 41 in ef9ee267. Bare bump of the chart's embedded `config:` block
and the README example; the chart does not mirror the read_before_write
section, so no field changes are needed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(agents): rewrite Responses and v1 content-block arguments too
Review finding on #5329 (P2): the content rewriter only handled Anthropic
`tool_use` blocks. With `use_responses_api=true` and
`output_version='responses/v1'`, AIMessage.content carries `function_call`
blocks whose `arguments` still hold the full write payload, and
langchain_openai's Responses input builder emits that block instead of the
rewritten structured call whose `call_id` it already carries. Standard `v1`
`tool_call` blocks likewise keep `extras.arguments`, which the v1->Responses
translator prefers over the structured args. So the blocked payload was
still sent on every later Responses API request.
`tool_call_args` now rewrites every content dialect that carries its own
copy of the arguments: Anthropic `tool_use` (input, drop partial_json),
Responses `function_call` (arguments, matched by call_id, `fc_...` item id
and status preserved), and v1 `tool_call` / `tool_call_chunk` (args plus
`extras.arguments`). Tests assert against the real adapter serializers:
`_construct_responses_api_input` for responses/v1, v1, and v0 messages,
`_convert_message_to_dict` for chat completions, and Anthropic
`_format_messages` for native and v1 content, plus an end-to-end probe
through the gate's wrap_model_call.
* fix(agents): pair blocked writes per call occurrence and defeat Responses chaining
Two review findings on #5329:
- Tool-call ids may repeat across assistant turns (DanglingToolCallMiddleware
pairs them with per-id queues). The gate matched blocked results against a
history-wide id set, so a successful write sharing an id with a later (or
earlier) blocked one also lost its payload and was labelled as blocked.
`_blocked_call_occurrences` now pairs ToolMessages with call occurrences
the same FIFO-per-id way and the selector keys on (message, call id).
- With `use_previous_response_id`, the OpenAI adapter sends only the messages
after the last AIMessage carrying a `resp_` response id and lets the server
rebuild the rest from its stored copy, which still holds the original
arguments and cannot be edited; every later response chains back to it.
`rewrite_messages_tool_call_args` now drops every `resp_` id from the
model-bound copy whenever it rewrote anything, so the adapter replays the
full rewritten history (the `use_previous_response_id=False` request
shape). OpenAI bills chained input tokens as input either way, so replay
costs no more; the state keeps its ids.
Tests cover success-before-block and block-before-success histories through
the Chat Completions serializer, and chaining through
`ChatOpenAI._get_request_payload` with `use_previous_response_id=True`:
unrewritten history chains and omits the call, rewritten history is replayed
with the placeholder and no `previous_response_id`.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix: preserve actionable subagent acceptance gaps
Distinguish completed execution from acceptance in delegation guidance. Retain bounded unmet and unverified criteria after compaction and guide the lead to address remaining work within its budget.
* docs: keep acceptance guidance within instruction budget
* fix(agents): prioritize loop hard stops across tool batches
Scan an admitted multi-tool response completely before selecting a soft warning, so any configured hard limit can reject the whole batch. Preserve warning priority and sliding-window accounting.
Add counter-level plus sync and async compiled-agent regressions proving rejected tools are not executed.
Fixes bytedance/deer-flow#5243. AI-assisted implementation and tests.
* fix(agents): rearm loop warnings after cross-tool eviction
When another tool evicts an older tool below its frequency warning threshold, clear the older suppression mark so a later burst can warn again.
Add the cross-tool sliding-window regression from the final boundary review. AI-assisted implementation and tests.
* test(agents): cover override-aware loop warning rearm
Cache the default frequency thresholds for sliding-window eviction and verify that an evicted tool uses its configured override when warning eligibility is rearmed.
Document that simultaneous frequency warnings preserve legacy first-crossing selection while hard stops remain batch-severity-first.
Addresses review on #5245. AI-assisted implementation and tests.
---------
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
