Otavio Rodrigues Santana 27cb73659d
fix(auth): correct OAuth conflict error message + validate multi-worker Postgres claim with real concurrency benchmark (#5026)
* fix(auth): correct OAuth uniqueness error and index parity on Postgres

create_user() caught any IntegrityError on commit and always reported it
as a duplicate email. The email pre-check already rules out a real email
collision in the common case, so any IntegrityError reaching that handler
is actually idx_users_oauth_identity firing instead -- confirmed against
both backends: SQLite reports "UNIQUE constraint failed:
users.oauth_provider, users.oauth_id", Postgres reports a
UniqueViolationError naming the same index. The caller saw "Email already
registered" for an OAuth account conflict, which is wrong and would send
API consumers debugging the wrong field.

Distinguish the two cases via a substring check on the driver error text
(both backends name the oauth columns) and raise an accurate message for
each.

Also add postgresql_where to the same index, alongside the existing
sqlite_where. This is not a correctness fix -- verified empirically that
Postgres already enforces the same practical uniqueness without it
(NULL is never equal to NULL in either backends unique index, so real
duplicate (provider, id) pairs are already rejected and NULL/NULL rows
are already unconstrained). postgresql_where makes the index genuinely
partial on Postgres too, matching the stated intent in the surrounding
comment and keeping the index smaller as the common case (plain-password
accounts, both columns NULL) accumulates.

* test(bench): add multi-process SQLite vs Postgres concurrency benchmark

CONFIGURATION.md documents that multi-worker deployments must use Postgres
because "SQLite silently ignores row-level locks", but nothing in the repo
exercised that claim against real separate worker processes -- the existing
checkpoint benchmarks (scripts/benchmark/checkpoint/) measure single-process
read/write latency, and the existing Postgres tests
(test_pg_schema_integration.py, test_multi_worker_postgres_gate.py) cover
schema placement and config validation, not throughput or lock behavior
under concurrent load.

run_concurrency_bench.py spawns N real OS processes (subprocess.Popen, not
asyncio tasks or threads within one process) against the shared users
table, mixing reads (get_user_by_email) and writes (create_user) at a
configurable ratio, and reports throughput, error counts by exception
type, and p50/p95/p99/max latency per run.

Measured locally (2/4/8/16 workers, 100 ops/worker, 70/30 read/write):
SQLite completed all operations with zero errors at every worker count
(busy_timeout absorbs contention rather than raising), but total
throughput stayed flat around 28-34 ops/s regardless of worker count, and
p99 latency grew from ~400ms at 2 workers to ~5.9s at 16, with a 22s max.
Postgres throughput scaled with worker count (41 to 66 ops/s) and p99
stayed under 500ms at every worker count tested. Raw JSON output from
both runs is available on request; exact numbers will vary by machine and
are not asserted in the test suite.

test_bench_concurrency.py unit-tests the pure aggregation logic
(percentile math, error grouping, crashed-worker handling) the same way
test_bench_checkpoint_channels.py does for the existing benchmarks --
fast, no DB required, not the full multi-process sweep in CI.

* fix(auth): inspect the driver exception for OAuth conflict detection

str(exc) embeds the full failed INSERT statement, whose column list
always names oauth_provider/oauth_id, so a substring check on it
misclassified every commit-time IntegrityError on the users table as
an OAuth conflict (reproduced on SQLite: a duplicate primary key with
a different email raised "OAuth account already linked: None/None").

_is_oauth_identity_violation now inspects exc.orig instead: constraint_name
on Postgres, both violated column names present (not a bare "oauth"
substring) on SQLite.

Also ships the alembic revision idx_users_oauth_identity's postgresql_where
predicate needed: 0001_baseline created it as a full index on Postgres,
and ORM metadata changes only affect fresh create_all databases, never an
already-versioned deployment.

Addresses review feedback from willem-bd.

* fix(bench): run the concurrency benchmark in an isolated schema and derive paths from the checkout

--pg-url accepted an arbitrary database URL while the code pinned
postgres_schema="public" and unconditionally ran DELETE FROM users --
against any non-disposable database that permanently destroyed every
auth account. Each run now generates a unique throwaway schema
(bench_<uuid>), points both the seeder and every worker subprocess at
it via postgres_schema, and drops only that schema (DROP SCHEMA ...
CASCADE) once the full worker-count sweep finishes.

Also stopped hard-coding /opt/deer-flow/backend as the checkout path
and .venv/bin/python3 as the interpreter: BACKEND_DIR is now derived
from Path(__file__), and workers are spawned with sys.executable (the
orchestrator's own interpreter) instead, so the documented
uv run python scripts/benchmark/concurrency/run_concurrency_bench.py
command works from any checkout.

Addresses review feedback from willem-bd.

* fix: shorten oauth-index revision id, repin migration-head assertions, fix bench read/write mix

- 0017_users_oauth_identity_partial_pg (36 chars) exceeded
  alembic_version.version_num's VARCHAR(32) limit, which would fail
  stamping/upgrading on both fresh and existing Postgres deployments.
  Renamed to 0017_oauth_identity_pg_partial (30 chars).
- Repinned every test asserting 0016_subagent_batches as the migration
  head (test_persistence_bootstrap[.py|_concurrency.py|_regression.py],
  test_migration_0004/0007/0015) to the new 0017 revision id.
- worker.py's `(i % 100) < int(read_ratio * 100)` assumed n_ops >= 100;
  at the documented default (50 ops/worker, 0.7 read ratio) it produced
  either all-reads or all-writes, never the claimed mixed workload.
  Replaced with read_count()/is_read_op(), which distribute an exact
  round(n_ops * read_ratio) reads evenly across the sequence via modular
  spacing, and added test_bench_worker.py covering the default values
  plus small op counts.

* fix(bench): establish a real physical connection before timing ops

async with sf(): pass entered an empty AsyncSession without checking out
a physical connection -- SQLAlchemy stays lazy until the first statement
executes. That pushed connection-establishment cost onto each worker's
first timed operation instead of conn_time_s, and at 16 workers those 16
cold first-ops (1% of a 1600-op sample) could skew the reported p99.
Execute a real `SELECT 1` before starting the timer instead.

Verified with a real end-to-end run (uv sync + sqlite backend, 2
workers/10 ops, 0 errors) plus the full auth/bench/migration-bootstrap
suites (135 tests) and ruff check/format, all clean.

* fix(bench): synchronize workers before timing, fix percentile off-by-one

Two remaining measurement issues from review:

- run_workers() started the wall clock before spawning any worker, so
  throughput/wall_time absorbed N processes' staggered Python-startup and
  connection-establishment cost, and early workers could run ahead of ones
  still starting. Workers now print READY right before their timed loop
  and block on stdin for a GO signal; the orchestrator waits for every
  READY, then starts the timer and releases all workers together.

- summarize()'s pct() used int(len(latencies) * p) directly as a
  zero-based index -- a one-based-rank-as-index bug that put p95 and p99
  at the same slot (the max) for any 20-or-fewer-sample run, and for the
  documented 100-sample default. Now delegates to
  checkpoint_bench_common.percentile(), the already-correct nearest-rank
  implementation used elsewhere in the same benchmark family, instead of
  a second, broken one.

Verified: 14/14 unit tests pass (2 new pinned-value regression tests for
the percentile bug, using the reviewer's own 20-sample repro), ruff
clean, and a real 2/4-worker SQLite multi-process smoke run completes
with distinct p95/p99/max latencies and no hang.

* fix(bench): absolute SQLite bench path, surface crash diagnostics, exit nonzero on failure; share OAuth index constant + cover Postgres branch

Three more findings from review at 5fd25a7:

- seed_baseline() cleaned an absolute .deer-flow/bench_data path but
  handed DatabaseConfig a relative one, which resolves against the
  CALLER's CWD -- not BACKEND_DIR. Invoking the documented command from
  anywhere other than backend/ silently pointed the seeder and the
  (cwd=BACKEND_DIR) workers at two different directories: workers crashed
  with 'unable to open database file' while the run still printed a
  well-formed summary and exited 0. Both seed_baseline() and worker.py's
  make_session_factory() now use the same absolute path.

- Crashed workers' stderr was captured then discarded, and main() always
  exited 0 -- an all-crashed sweep was indistinguishable from a real
  (uneventful) measurement to anything checking the exit code or
  --out. run_workers() now prints each crash immediately and tags it with
  the real worker_id (previously always None); summarize() exposes
  crashed_worker_errors alongside the existing crashed_workers count;
  main() exits 1 via the new summary_indicates_failure() whenever any
  sweep crashed or fell short of expected_total_ops.

- idx_users_oauth_identity was hardcoded separately in the ORM Index and
  in _is_oauth_identity_violation's Postgres branch, with no test to
  catch drift, and that branch had zero non-skipped coverage (its only
  guard needs a live Postgres CI never configures). Exported
  OAUTH_IDENTITY_INDEX_NAME from user/model.py as the shared source of
  truth (migrations intentionally keep their own frozen literal, matching
  every other revision in that package) and added stub-exception unit
  tests pinning both the asyncpg constraint_name path and the sqlite
  message-substring path, positive and negative.

Verified: 107 passed locally (auth + bench-unit suites), ruff clean, and
two real reproductions -- invoking run_concurrency_bench.py from a
scratch directory outside backend/ (the reviewer's exact repro) now
completes 8/8 ops with crashed_workers: 0 instead of crashing, and the
new crashed_worker_errors/exit-code logic is exercised directly by the
new unit tests against the real summarize()/summary_indicates_failure().

* fix(auth): attribute create_user IntegrityErrors to the right constraint

Two coupled review findings on the classification helpers:

P3 (fall-through) -- after ruling out the OAuth-identity index, create_user
raised "Email already registered: {email}" for every remaining
IntegrityError, including the duplicate-primary-key case the new
regression test exercises, whose address is not registered. Added
_is_email_violation() so the email message is used only for an actual
users.email collision that raced past the pre-check; anything else (in
practice a duplicate id) now raises a neutral
"User already exists (constraint: <name>)".

P2 (unreachable asyncpg branch) -- exc.orig is not the asyncpg error.
SQLAlchemy's asyncpg dialect re-raises its own DBAPI IntegrityError
(pgcode/sqlstate only) 'from' the real asyncpg error, so constraint_name
lives on exc.orig.__cause__. getattr(exc.orig, "constraint_name", None)
was always None on Postgres; the helpers only worked there by accident,
matching asyncpg's DETAIL line in the message fallback. Added
_driver_constraint_name() which walks orig then orig.__cause__, and the
stub tests now model that real shape (orig wrapper + __cause__) instead of
a constraint_name that no driver puts on orig directly.

Tests: 76 passed. New coverage for the email-race path, both new helpers
on each backend, the neutral fallback message, and the cause-chain walk.

* fix(bench): match app SQLite PRAGMAs in workers; fail a sweep on any op error

Two review follow-ups:

- worker.py opened its SQLite engine with only connect_args timeout=30.
  synchronous and foreign_keys are per-connection PRAGMAs, so workers ran
  at SQLite's synchronous=FULL / foreign_keys=OFF while a real Gateway
  worker runs synchronous=NORMAL (persistence/engine.py::_enable_sqlite_wal)
  -- an extra fsync per commit on the measured 30%-write path, overstating
  SQLite's cost in the direction that flatters the "use Postgres"
  conclusion. Added a connect listener applying the same four PRAGMAs, with
  a test asserting synchronous/foreign_keys/journal_mode on a real worker
  connection.

- summary_indicates_failure() only looked at crashes and the completed vs
  expected op counts, so a sweep where every op completed but raised
  (e.g. writes hitting OperationalError) passed as a clean measurement:
  completed_ops == expected, 0 crashes. Added an "errors > 0" clause; the
  error breakdown stays in the JSON, only the exit code changes. Test added.

test_bench_concurrency.py + test_bench_worker.py green (20), plus a real
2-worker sqlite smoke run (6/6 ops, 0 errors, exit 0).

* fix(auth): match the real email index name; only claim "exists" for uniqueness

Review follow-ups on the classification helpers:

- email is mapped_column(unique=True, index=True), which SQLAlchemy and
  0001_baseline realise as a single UNIQUE INDEX (ix_users_email), not a
  named UNIQUE constraint. _is_email_violation compared the driver
  constraint name against "users_email_key", which Postgres never emits,
  so that arm was dead on Postgres (SQLite matched via the message). Fixed
  to ix_users_email.

- the residual IntegrityError fallback raised "User already exists" for
  every remaining IntegrityError -- a NOT NULL / CHECK / foreign-key
  violation is not a "user already exists" condition and is not part of
  create_user's ValueError contract. Added _is_uniqueness_violation
  (sqlstate 23505, or the SQLite "UNIQUE/PRIMARY KEY constraint failed"
  message); only that raises the "already exists" ValueError, everything
  else propagates unchanged.

- documented scripts/benchmark/concurrency/ in backend/AGENTS.md alongside
  the other benchmark family.

Tests: 78 auth + 20 bench-unit pass, ruff clean. New coverage for
_is_uniqueness_violation on both backends and for a non-uniqueness
IntegrityError propagating out of create_user.

* fix(bench): don't pre-close worker stdin (breaks communicate); require --pg-url for postgres

* fix(bench): ruff format; time throughput on the op phase, not teardown

- lint-backend: ruff format the files touched in this PR.
- Throughput window (P2): the orchestrator sampled its wall clock after
  every worker's communicate() returned, so it also covered each worker's
  engine.dispose(), result serialization and stdout transfer. Each worker
  now times just its operation phase (GO -> last op) and reports
  ops_elapsed_s; summarize() uses max(ops_elapsed_s) over the workers -- all
  released by the same GO -- as the throughput window (ops_window_s).
- Exercise migration 0018 (P2): test_user_oauth_partial_index.py goes
  through bootstrap create_all(), which builds the partial index from ORM
  metadata and never runs 0018.upgrade(). New Postgres-gated
  test_migration_0018_oauth_identity_pg_partial.py alembic-upgrades to 0017
  (full index), then 0018 (asserts the predicate appears), then downgrades
  (asserts the full index is restored) and re-upgrades.

* docs(middlewares): tighten SandboxAudit and Clarification entries in AGENTS.md

PR #5134 grew agents/middlewares/AGENTS.md ~1.8 KB, pushing the effective
AGENTS.md chain for that directory over the 96 KiB hard limit once this
branch also documents scripts/benchmark/concurrency/ in backend/AGENTS.md.
Condense the two longest middleware entries (SandboxAuditMiddleware,
ClarificationMiddleware) without dropping any identifier, example, issue
reference, ordering constraint, or documented gap; chain back to ~96.8 KiB.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-03 08:17:26 +08:00

35 KiB
Raw Blame History

Middleware Chain

Persisted delegation verdicts are untrusted durable context; ledger rendering revalidates them and ignores malformed values.

Lead-agent middlewares are assembled in strict order across three functions: the shared base in packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py (_build_runtime_middlewares, exposed via build_lead_runtime_middlewares), then the lead-only middlewares appended in packages/harness/deerflow/agents/lead_agent/agent.py (build_middlewares). Items marked (optional) are appended only when their config/runtime condition holds, so the live chain length varies.

Message provenance. A middleware that injects or rewrites a message stamps additional_kwargs with the neutral provenance keys from deerflow_extension_api.provenance (deerflow_content_kind, deerflow_producer_kind, and optionally deerflow_producer_entity_id) via provenance_kwargs(). The producer is not recoverable downstream — by the model-call boundary the message is indistinguishable from any other — so the fact is recorded where it is known. Stamping is unconditional: a fact whose presence depends on whether an observer is installed is not a fact. All three keys are in _SERVER_OWNED_MESSAGE_METADATA_KEYS, so a caller cannot forge provenance on inbound messages. Currently stamped by: DynamicContext (reminder + memory), DurableContext (contract + data), SystemMessageCoalescing, ViewImage, SkillActivation. Summarization, Title, and Memory are deliberately absent: Summarization's and Title's own model calls are already attributed through system-model-call observation (SystemOperationKind.SUMMARIZATION / .TITLE), and the summary text they produce only ever enters a request via DurableContextMiddleware's already-stamped durable_context_data block — there is no separate message of theirs to stamp. Memory only reads messages to queue them for extraction; the recalled-memory content that actually re-enters context is DynamicContext's dynamic_context_memory stamp, not anything Memory itself produces.

Middleware self-description. A middleware whose configuration changes agent behaviour implements release_policy_parameters() -> dict[str, object] (deerflow_extension_api.release.ReleasePolicyProvider, duck-typed — no base class). Values must be JSON-serialisable; long text is hashed with canonical_hash rather than embedded, because a declaration is an identity and not a copy of the prompt. collect_release_policies() gathers them from an assembled stack. Adding a behaviour-affecting field to a middleware means adding it to that middleware's declaration in the same change.

Shared runtime base (build_lead_runtime_middlewares; subagents reuse most of this via build_subagent_runtime_middlewares):

  1. InputSanitizationMiddleware - First, so it is the outermost wrap_model_call wrapper; every inner middleware (including LLM retries) sees sanitized messages. additional_kwargs.original_user_content is server-owned provenance: Gateway strips caller-supplied values for non-internal run requests, trusted IM calls may carry the string they captured before adding transport/file context, and the middleware replaces any non-string value before wrapping. Uploads and sanitization retain first-writer-wins only for validated strings.

  2. ToolOutputBudgetMiddleware - Caps tool output size (per app config) before it re-enters the model context. Oversized results are externalized to tool_output.storage_subdir (default .tool-results, shared constant TOOL_RESULTS_DIRNAME) under the thread outputs dir with a typed synopsis + read_file reference left in context; those files are process feedback, so the workspace-changes scanner excludes that directory and run delivery verification never counts them as produced artifacts

  3. ToolResultSanitizationMiddleware - Neutralizes framework/injection tags (e.g. <system-reminder>) and boundary markers in remote-content tool results (web_fetch/web_search/image_search/web_capture) so attacker-controlled fetched pages cannot forge trusted framework context. Mirrors InputSanitizationMiddleware's user-input guardrail for the other untrusted-content entry point; sits inner of ToolOutputBudgetMiddleware (neutralizes the raw output, then the budget truncates). Local tool output (bash/read_file) is left untouched. Scope is a name-based allowlist for the first-party web tools, plus every MCP-sourced tool via its deerflow_mcp metadata tag, so an MCP server naming its fetcher fetch_url is still covered

    Result-rewriting middlewares between the raw callable boundary and the model-visible result append a declared entry to additional_kwargs["deerflow_tool_transforms"] via agents/middlewares/tool_transform_meta.py::append_tool_transform. The trail is ordered by application — the last entry produced the final visible bytes — so an observer classifies raw→visible transforms from facts rather than by sniffing output wording.

  4. ThreadDataMiddleware - Creates per-thread directories under the user's isolation scope (backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}); resolves identity via resolve_runtime_user_id(runtime), including Gateway runtime context and standalone LangGraph Server auth, then falls back to the request ContextVar / "default"

  5. UploadsMiddleware - Tracks and injects newly uploaded files into conversation (lead agent only); upload existence checks use the same runtime-resolved user bucket as thread-data creation

  6. SandboxMiddleware - Acquires sandbox, stores sandbox_id in state. The lead runtime normally owns the thread's physical Agent-skill projection; delegated subagents and the prompt-only bootstrap agent are non-owners, so their narrower discovery allowlists never rebuild the shared thread view or force eager sandbox acquisition.

  7. DanglingToolCallMiddleware - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., user interruption), preserving raw provider tool-call payloads in additional_kwargs["tool_calls"]; malformed tool-call names and arguments are sanitized in the model-bound request so strict OpenAI-compatible providers do not reject the next request

  8. LLMErrorHandlingMiddleware - Normalizes provider/model invocation failures into recoverable assistant-facing errors before later stages run

  9. Authorization / GuardrailMiddleware - Up to two independent pre-tool-call gates run here. When authorization.enabled, the AuthorizationProvider instance already used for Layer 1 capability filtering is wrapped by GuardrailAuthorizationAdapter and reused for Layer 2 execution checks. A generated tool_search bypasses the adapter's second provider call only when the current build has a concrete deferred setup; its catalog was already filtered by Layer 1, and an ordinary same-named tool without that deferred setup receives no exemption. When guardrails.enabled, the explicitly configured GuardrailProvider is appended after authorization and still evaluates every call, including tool_search. Authorization therefore runs outermost and can deny before an external guardrail call; both use the existing middleware's fail-closed, audit, sync/async, and error-ToolMessage behavior. See the authorization RFC and docs/GUARDRAILS.md.

    Every guardrail decision path publishes a neutral deerflow.authz.outcome.AuthorizationOutcome into the per-run runtime context, keyed by tool_call_id under the __-prefixed __authorization_outcome key (so build_run_config strips caller-supplied forgeries). Consumers pop it; the publisher and the consumer share only that contract module.

  10. SandboxAuditMiddleware - Audits sandboxed shell/file operations before tool execution; command classification is defense-in-depth and audit, not a security boundary (the sandbox is the isolation boundary). Command substitution is judged by position, not the presence of $(: command position ($(curl url), `curl url`, the word after |/&&/;, an eval/source argument) executes fetched content and is blocked; value position (x=$(curl url), echo $(curl url), an argument, a for word list) only captures output and passes (#4611). So _HIGH_RISK_COMMAND_POSITION_PATTERNS is matched anchored against each sub-command from _split_compound_command(split_pipes=True), never the whole string; pipe-spanning rules (| sh, base64 -d | ...) still use _classify_command's whole-command Pass 1. _COMMAND_POSITION_PREFIX extends the anchor over leading assignments and exec wrappers (FOO=1 $(curl url), env/command/builtin/exec/nohup/time/sudo/doas); its assignment branch requires whitespace before the substitution, which keeps x=$(curl url) in value position. Two contexts are deliberately position-blind (matched whole-command in Pass 1, since they execute their input anywhere, e.g. xargs sh -c "$(curl url)"): an eval/source argument, and an interpreter code-string flag — -c (shells, python), -e (perl/ruby/node), -p (perl/node), -r (php) — plus the here-string (<<<) reaching the same place via stdin. All three substitution spellings ($(, <(, `) share one _RISKY_SUBSTITUTION opener. An unquoted newline splits like ; (else echo hi\n$(curl url) evades the anchored rules). Heredoc bodies are data: _split_compound_command records headers (<<EOF, <<-EOF, <<'EOF') and consumes their bodies verbatim, so a body line starting $(curl url) isn't promoted to a command position; <<< (here-string, needs look-ahead + look-behind) and a << inside $(( ))/(( )) (bit shift, arithmetic depth tracked with the quote flags) must not open one. This is a heuristic, not shell parsing — an unterminated body consumes the rest of the string, an unclosed (( only disables heredoc detection, and the failure direction is always toward more command positions, not fewer. Known gaps: process substitution outside eval/source (. <(curl u)) is undetected, and two-step forms (x=$(curl u); eval "$x") need dataflow analysis. No config gate — appended unconditionally in _build_runtime_middlewares, for both lead and subagents.

  11. ReadBeforeWriteMiddleware - (optional, if read_before_write.enabled, default on) Outermost write gate (issue #3857): read_file stamps a content hash onto its ToolMessage; write_file (append/overwrite-existing) and str_replace are blocked unless the newest mark for that path matches the file's current hash. Sits outside ToolProgressMiddleware and ToolErrorHandlingMiddleware so a blocked write returns immediately without consuming a ToolProgress slot. Blocked results call normalize_tool_result directly to stamp deerflow_tool_meta (recoverable_by_model=True) before returning, keeping the result well-formed for any outer consumer. Marks live on messages, so summarization dropping the read result invalidates the gate automatically; writes never refresh marks, forcing a re-read between consecutive edits. Gate check + tool execution are serialized per (thread, path) so same-turn parallel writes cannot reuse one stale mark; on sandboxes whose read_file reports failures as "Error: ..." strings instead of raising (AIO/E2B), uninspectable targets fail open (creation proceeds, no mark stamped). The middleware also owns the sandbox authorization scope for these composed calls: pre-write inspection, the tool body, and post-read hashing share one sync/async provider decision, while SandboxAuthorizationError bypasses the generic inspection fail-open paths and becomes an error ToolMessage.

  12. ToolProgressMiddleware - (optional, if tool_progress.enabled) State-machine-based stagnation guard (RFC #3177). Outer wrapper around ToolErrorHandlingMiddleware so its wrap_tool_call receives results already stamped with deerflow_tool_meta. Tracks per-(thread, tool) consecutive "no-new-info" calls across three error categories: (a) recoverable_by_model=True (no_results, not_found, permission, Jaccard-duplicate success): ACTIVE → WARNED (terminal — hint re-injected on each subsequent problem); (b) recoverable_by_model=False, action≠stop (rate_limited, transient): ACTIVE → WARNED → BLOCKED after warn_escalation_count more problems; (c) recoverable_by_model=False, action=stop (auth, config, internal): immediately BLOCKED on first occurrence. Division of labor with LoopDetectionMiddleware: ToolProgressMiddleware is a result-quality guard — fires after tool execution and blocks specific tools that stop producing new information; LoopDetectionMiddleware is a call-pattern guard — fires after the model responds and hard-stops the whole turn when the model repeatedly issues identical tool_calls. Both can inject HumanMessage hints in the same model call without conflict; neither reads the other's internal state.

  13. ToolReceiptMiddleware + ToolErrorHandlingMiddleware - ToolReceiptMiddleware is (optional, if verification.receipts_enabled, default on). It is the outermost wrap_tool_call layer — registered ahead of entries 9-12 — because Guardrail/SandboxAudit/ReadBeforeWrite/ToolProgress can short-circuit a call with their own ToolMessage (and SandboxAudit rebuilds medium-risk results); an inner receipt layer would silently gap the ledger on those results (ordering constraints in deerflow.extensions.ordering). Normal results still carry the deerflow_tool_meta status ToolErrorHandlingMiddleware stamps on the inner return path; short-circuit messages self-stamp meta or fall back to message.status. It stamps deterministic provenance (tool name, status, args/output hashes, byte count, timestamp) onto direct ToolMessage results and every matching ToolMessage carried in Command.update.messages, including delegated task, present_file, view_image, and tool_search results; before model calls it derives a hidden receipt ledger (display ids r1..rN) from message state, and when the 2,000-character budget is exceeded the newest receipts are retained in chronological order with their original ids plus an older-receipts omission marker. Rendering returns both the text and its retained receipt subset; every response that received a ledger carries only that exact server-owned subset, never omitted receipts. Snapshot validation accepts a strictly consecutive positive original-id range (for example r24–r30) rather than requiring r1, so subagent terminal citation verification resolves ids against evidence present in the citing turn even when later summarization drops and renumbers tool messages. Model-generated citation IDs are digit-bounded before integer conversion; oversized IDs are ignored as malformed input rather than raising through task write-back. Citation parsing deduplicates exact (id, anchor) pairs, not IDs alone, so repeated identical references stay compact while every distinct anchor claim is verified. Gateway strips delegated receipts/verdicts from external messages. ToolErrorHandlingMiddleware receives AppConfig, converts tool exceptions into error ToolMessages so the run can continue instead of aborting, stamps every result with deerflow_tool_meta (status / error_type / recoverable_by_model / recommended_next_action / source) via tool_result_meta.normalize_tool_result, stamps structured metadata for task exception wrappers, and stamps skill-read metadata for downstream durable-context capture. Task tool result text is generated from the same status/result/error inputs as the structured metadata so callers do not hand-write a second protocol string.

Authorization identity plumbing is independent of whether authorization enforcement is enabled. Gateway removes client-supplied is_internal / authz_attributes / channel_user_id, derives is_internal only from the server-owned request.state.auth_source, and accepts channel_user_id only from an internally authenticated IM caller's top-level body.context; free-form body.config can never supply it. build_principal_from_context is the shared Principal builder for assembly-time authorization and GuardrailAuthorizationAdapter; it applies default_role, strict-boolean internal provenance, and copy-on-read authz_attributes. The built-in RBAC provider validates authorization.default_role during provider resolution so an unknown fallback role fails agent construction instead of degrading into an empty tool set. Task delegation carries is_internal plus copied attributes through SubagentExecutor, while GuardrailMiddleware maps the same runtime fields into GuardrailRequest. Phase 1B applies Layer 1 before deferred-tool assembly on the lead, native-subagent, and embedded-client paths, then passes the same provider instance into Layer 2. Framework-provided describe_skill and memory tools are included in Layer 1 but restored to their legacy post-tool_search ordering afterward. DeerFlowClient.stream() treats its in-process caller as trusted and accepts the same identity fields as keyword overrides; it includes the complete Principal in its agent cache key and deep-copies nested attributes so caller mutation cannot make a stale tool set look current.

Gateway route authorization uses authz.py::resolve_route_permissions() as the single provider integration point for both AuthMiddleware and decorator-only authentication. When enabled, it evaluates the six registered threads:* / runs:* permissions as resource="route" requests whose targets are the full resource:action strings. Decisions use the async provider API and are cached for the request in AuthContext; decorators do not call the provider again. Provider resolution or decision errors follow authorization.fail_closed, scoped per permission for decision errors. When authorization is disabled, the legacy complete permission set is returned without resolving a provider. Existing owner_check enforcement and require_admin_user() management gates remain independent and unchanged. Tests: tests/test_authorization_route_permissions.py, tests/test_auth.py, and tests/test_auth_middleware.py.

Model authorization uses authz.py::resolve_model_authorization() (same cached-provider, internal-role, and principal-building path as route authorization) as the Gateway integration point for the models router: list_models filters names through filter_resources(principal, "model", ...), and get_model enforces authorize(resource="model", action="use") with a deny surfacing as 403; provider errors follow authorization.fail_closed (fail-open returns the unfiltered list / proceeds). At runtime, lead_agent/agent.py::_authorize_model_name — called from _make_lead_agent and from DeerFlowClient._ensure_agent — applies the same model:use check to the resolved model name. On deny it scans the filter_resources-visible names (excluding the denied model), re-verifying each candidate with authorize("model", "use") before falling back, because a custom provider may allow list while denying use; no usable fallback raises under fail_closed and keeps the original model under fail-open. The built-in RBAC provider maps this to the per-role models policy key. Tests: tests/test_models_authorization.py.

Sandbox authorization (sandbox:execute) gates every sandbox acquisition before provider.acquire — including this middleware's eager path (before_agent / abefore_agent skip acquisition on deny instead of raising, deferring to the lazy per-tool gate). See the sandbox module guide authorization-gate paragraph and tests/test_sandbox_authorization.py.

Before changing a later authorization phase, read the authorization RFC and its implementation notes. The notes are the cumulative handoff record for merged PR behavior, reviewer feedback, trust-boundary decisions, deferred scope, and required regression coverage.

Lead-only middlewares (build_middlewares, appended after the base):

  1. DynamicContextMiddleware - Injects the current date (and optionally memory) as a <system-reminder> into the first HumanMessage, keeping the base system prompt fully static for prefix-cache reuse
  2. SkillActivationMiddleware - Detects strict /skill-name task syntax on the latest real user message, resolves only enabled and runtime-allowed skills, injects the SKILL.md body as hidden current-turn context, and records a middleware:skill_activation audit event
  3. SkillToolPolicyMiddleware - Applies allowed-tools only after real activation; passive enabled skills and a custom agent's configured skill allowlist do not clamp the lead toolset. A run-scoped slash activation is authoritative and suppresses skill_context as a policy source, so reading another skill cannot widen the explicit skill's tools; without slash activation, skills captured after configured read_file loads retain the existing union semantics. The middleware filters model-visible schemas and blocks unauthorized execution, resolving canonical paths against the live enabled/agent-allowed registry on every model call, then stores a versioned, JSON-safe, middleware-token-bound decision signed by policy source plus active paths in run context for the resulting tool calls to reuse. The next model call always refreshes it, and malformed, foreign, stale, or unmatched decisions fall back to live resolution. tool_search and describe_skill remain framework-safe discovery tools under a restrictive policy; they may reveal or promote metadata, but a deferred business tool must still be declared by the active policy before its schema or execution can survive the policy middleware. The decision's owner token is authorization-sensitive, so its reserved context key is owned by runtime.secret_context and included in REDACTED_CONTEXT_KEYS for observable and persisted context copies. Registry load failures and a non-empty active set with no authorized skill fail closed to framework-safe tools; an individual stale path is skipped only when at least one valid active skill remains. This is best-effort behavioral scoping rather than a hard security boundary: alternate loads such as bash cat are not captured, and bounded autonomous skill_context can evict old entries. task is not framework-exempt, so a restricted skill cannot delegate around its policy. The middleware must remain immediately after SkillActivationMiddleware (which publishes the slash source through runtime.secret_context's public path helpers authenticated by a required token shared only within the assembled middleware chain) and immediately before DurableContextMiddleware; assembly and compiled-graph tests pin ordering, token sharing, schema filtering, and execution blocking.
  4. DurableContextMiddleware - Captures task delegations into ThreadState.delegations (including in-progress dispatches and terminal result summaries) and loaded skill-file references (name/path/description, parsed in-memory - not the body) into ThreadState.skill_context before summarization can compact the paired tool-call/result messages, then projects durable context into each model request. Static authority rules are injected as a SystemMessage; untrusted field values (summary_text, delegation results, skill descriptions) are injected separately as a hidden HumanMessage data block so compressed history, delegated work, and which skills are active stay visible without being stored as messages or promoted to system-role instructions. build_subagent_runtime_middlewares also attaches this middleware immediately before subagent summarization so a compacted summary_text is projected ahead of a preserved assistant/tool tail instead of leaving strict providers with an assistant-first request.
  5. SummarizationMiddleware - (optional, if enabled) Context reduction when approaching token limits. Compaction must preserve the latest real user request by exact message ID while allowing stale DynamicContext ID-swap __user peers to enter the summary; tagged dynamic-context reminders remain preserved. Do not move the cutoff backward to retain the current request, because that also keeps early AI/Tool turns active and can make first-turn long-analysis compaction a no-op. tests/test_summarization_middleware.py pins both the multi-turn stale-peer case and the first-turn long-analysis case.
  6. TodoListMiddleware - (optional, if is_plan_mode) Task tracking with the write_todos tool
  7. TokenUsageMiddleware - (optional, if token_usage.enabled) Records token usage metrics; subagent usage is read from terminal ToolMessage.additional_kwargs in the current run and merged back into the dispatching AIMessage by message position. The same state update marks the ToolMessage with subagent_token_usage_attributed=true, so checkpoint replay or middleware re-entry cannot add the cumulative snapshot twice; missing/malformed usage or a result with no matching dispatch remains unmarked and retryable.
  8. TitleMiddleware - Auto-generates the thread title after the first complete exchange and normalizes structured message content before prompting the title model. If a first-turn run is interrupted before this middleware can write a title, runtime/runs/worker.py keeps the run in a finalizing state, persists a local fallback title from the latest checkpoint or original run input, and then syncs it to threads_meta.display_name. Replacement runs admitted by multitask_strategy="interrupt" / "rollback" wait for older same-thread finalization before entering the graph; the interrupted run only skips the fallback title write once a later run has started and may have advanced the checkpoint.
  9. MemoryMiddleware - Queues conversations for async memory update (filters to user + final AI responses); captures the runtime-resolved user so standalone LangGraph Server reads and writes stay in the same bucket
  10. ViewImageMiddleware - (optional, if the model supports vision) Appends a hidden HumanMessage with base64 image data, identified by a reserved ID prefix plus a server-owned metadata marker, to ModelRequest.messages in wrap_model_call / awrap_model_call. The payload lives only in that request and is never returned as a state update, so no checkpoint carries it and an interrupted run cannot strand it in history; state keeps only the lightweight viewed_images metadata. It owns that context and rebuilds it per call: its own message is swept out of the request first — a thread checkpointed by the earlier before_model/after_model pair (which wrote the payload into state and took it back out with RemoveMessage) can carry one that reached state but was never removed, and leaving it in would resend that base64 in every later request for the life of the thread — then a freshly built one is appended when warranted. The sweep requires both the reserved ID prefix and the server-owned marker, so a client cannot get its own message dropped; unmarked leftovers predating the marker are left in place and merely not duplicated
  11. McpRoutingMiddleware - (optional, if tool_search.enabled and PR1 MCP routing metadata produce a routing index) Auto-promotes matching deferred MCP tool schemas before the model call by writing a minimal promoted state update. It matches only the latest real HumanMessage, uses the global tool_search.auto_promote_top_k limit (default 3, clamped to 1..5), never executes tools, and must be installed before DeferredToolFilterMiddleware
  12. DeferredToolFilterMiddleware - (optional, if tool_search.enabled) Hides deferred (MCP) tool schemas from the bound model until tool_search or McpRoutingMiddleware promotes them (reads per-thread promotions from ThreadState.promoted, hash-scoped)
  13. SystemMessageCoalescingMiddleware - Merges every SystemMessage into a single leading SystemMessage per request; provider-agnostic fix for strict backends (vLLM/SGLang/Qwen/Anthropic) that reject non-leading system messages. Touches the per-request payload only (checkpoint state unchanged); on midnight crossings only the latest dynamic_context_reminder SystemMessage survives. The subagent builder places its date-only context middleware immediately before this coalescer, so the built-in subagent prompt and hidden date reminder still reach providers as one leading system block
  14. SubagentLimitMiddleware - (optional, if subagent_enabled) Truncates excess ordinary task tool calls to enforce both the per-response concurrency limit (max_concurrent_subagents, resolved against startup subagent_runtime.max_running and the 1-64 safety range before construction) and the per-run total delegation cap (max_total_subagents runtime override or subagents.max_total_per_run, default 6, clamped to 1-50). The total cap counts current-run entries in the durable delegation ledger (entries are tagged with run_id when captured), so repeated planning checkpoints in one run cannot keep launching legal-sized batches indefinitely, while later user turns in the same thread get a fresh run budget. Explicit durable batch_task calls are a separate mode with persisted total/live/running limits and are not rewritten into ordinary ledger entries. If the ordinary cap is exhausted, the middleware strips remaining task calls, forces finish_reason="stop", and appends a visible limit note so the run can synthesize existing results instead of ending with an empty tool-call response.
  15. LoopDetectionMiddleware - (optional, if loop_detection.enabled) Detects repeated tool-call loops; hard-stop clears both structured tool_calls and raw provider tool-call metadata before forcing a final text answer; stamps loop_capped via consume_stop_reason (#3875 Phase 2), symmetric to TokenBudgetMiddleware; persists warned-state transitions (first per call hash or per tool-frequency burst) and hard stops as middleware:loop_detection, attributed with is_subagent and the optional agent_id, without tool arguments, message content, tool results, or argument-derived hashes. Ordinary task-tool subagents receive a loop-detection-only recorder proxy that forwards the append to the parent run loop; never pass the loop-bound RunJournal itself into their isolated event loop. Durable batch subagents have no parent run journal and do not persist these transitions
  16. TokenBudgetMiddleware - (optional, if token_budget.enabled) Enforces per-run token limits
  17. Custom middlewares - (optional) Any custom_middlewares passed to build_middlewares are injected here, before config-declared extensions and the terminal-response/safety/clarification tail
  18. Configured extension middlewares - (optional, if extensions.middlewares is set in config.yaml or extensions_config.json) Zero-argument AgentMiddleware classes loaded from module.path:ClassName entries via deerflow.reflection.resolve_class. Missing packages, invalid classes, and broken modules fail loudly at agent creation. These run after built-ins/programmatic custom middleware and after the lead/subagent loop/token guards, but before the terminal-response/safety/clarification tail; subagents receive the same configured extension middleware class list before their safety tail. Treat these files as trusted operator config because middleware paths instantiate arbitrary code. Gateway skill/MCP toggle endpoints preserve this field through to_file_dict() but must not add a write path for extensions.middlewares without an explicit trust-boundary review. Lead-only vs subagent-only middleware lists and per-context constructor parameters are not expressible in this MVP.
  19. TerminalResponseMiddleware - When a provider returns an empty terminal AIMessage after tool execution, injects a hidden recovery prompt and retries the model once; a second empty response is replaced in checkpoint state by a visible error fallback marked for the run worker, so the run finishes as an error instead of a silent success
  20. ModelLengthFinishReasonMiddleware - Records stop_reason=model_length_capped when provider-specific length detectors match a terminal AIMessage without tool-call intent (finish_reason=length / MAX_TOKENS, or stop_reason=max_tokens), preserving the original assistant content and never reparsing textual tool-call-like envelopes
  21. SafetyFinishReasonMiddleware - (optional, if safety_finish_reason.enabled) Suppresses tool execution when the provider safety-terminated the response (e.g. finish_reason=content_filter); registered after terminal-response/custom/configured middlewares so LangChain's reverse-order after_model dispatch runs it first
  22. ClarificationMiddleware - Intercepts ask_clarification, writes a readable ToolMessage.content fallback plus a structured ToolMessage.artifact.human_input payload, and interrupts via Command(goto=END) (must be last). after_model drops same-turn sibling tool calls so they cannot run before the user answers; a malformed ask_clarification parked on invalid_tool_calls is the same stop signal. disable_clarification runs keep the siblings. Payloads are versioned — legacy free_text/choice_with_other stay version: 1; the v2 form mode (from fields) is version: 2 so older frontends reject it and fall back to plain text. Field normalization is deterministic and lives in the middleware (it short-circuits before tool execution, so tool-arg typing gives no runtime validation), and it is atomic: any structurally broken entry — non-dict, bad/duplicate name, a name colliding with a JS Object.prototype member (__proto__/constructor), or exceeding the caps (16 fields / 24 options per field / 200 chars per text / MAX_FORM_SERIALIZED_BYTES = 16KB UTF-8, the per-item caps alone admitting forms whose IM text fallback overruns channel limits) — degrades the whole form to the legacy option/free-text modes, so a card never renders "complete" while missing a field. Benign issues degrade locally (unknown types — incl. unhashable JSON like type: [], which must not raise from the membership probe — and option-less selects become text); options are trimmed/deduped with blanks dropped (form- and top-level) since the frontend rejects blank labels. XML-to-dict option payloads are recursively flattened from dict/list containers in source order, scalar leaves kept, residual XML tags stripped before that trimming. Checkboxes are booleans defaulting to "no"; required on one means consent semantics. The response protocol is unchanged (v1 text/option): form cards submit a text summary as response_kind: "text", so journal persistence needs no new allowlist entries. Because this middleware can short-circuit before on_tool_end, RunJournal does a root-run reconciliation for ToolMessages whose tool_call_id came from the current run, so cards survive checkpoint compaction. That reconciliation is not ask_clarification-only — any middleware that answers a tool call has the same gap, and a result the user saw must not vanish on reload (#4666 — ReadBeforeWriteMiddleware blocked-write errors reached the UI but not the event store). It is bounded by three conditions, not a name allowlist: the message is user-visible, the call belongs to this run's lead agent (_remember_current_run_tool_calls records lead-agent calls only; subagent results stay in subagent.step), and it is not already persisted. Human Input Card replies are hide_from_ui HumanMessages with additional_kwargs.human_input_response; RunJournal persists only allowlisted hidden sources (currently ask_clarification) as llm.human.input.