github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-09-23 21:16:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(middleware): add deterministic PII redaction for model-bound context (#5527)

Browse Source 
* feat(middleware): add deterministic PII redaction for model-bound context

* fix(middleware): claim national IDs before cards, redact Command results, preserve ToolMessage fields

- Reorder detectors so checksum-gated national IDs run before the credit-card
  detector; an 18-digit resident ID whose digit run also passes Luhn is no
  longer mislabeled [CREDIT_CARD_n] (review finding, reproduced at 0a2a9d0)
- Redact ToolMessages carried in Command.update.messages, mirroring
  ToolResultSanitizationMiddleware's dc_replace pattern
- Rebuild redacted ToolMessages via model_copy so artifact and
  response_metadata survive
- Extend the numbered middleware chain in agents/middlewares/AGENTS.md

* fix(middleware): span one redactor per Command result; refresh stale AGENTS.md entry range

- Placeholder numbering now continues across every ToolMessage carried in a
  single Command result (one _Redactor per _redact_result call) instead of
  restarting per message
- The renumbered AGENTS.md chain still referenced entries 9-12 in the
  ToolReceiptMiddleware entry; it now reads entries 10-13

* docs(agents): trim PiiRedactionMiddleware entry to fit the AGENTS.md chain budget

The main merge (fb36e0e) pushed the effective middlewares chain to 98341
bytes, 37 over the 98304 hard limit checked by agent-guidance (AG002).
Compress the entry while keeping the load-bearing facts: config gate, both
interception points incl. Command coverage, detector order rationale,
per-result numbering continuity, irreversibility, memory follow-up.

* fix(middleware): redact compaction input and reinjected summaries; harden detectors

Review round 3 on #5527:
- [P1] SummarizationMiddleware invokes its summary model directly from
  before_model, outside PiiRedactionMiddleware's wrap_model_call, so raw
  thread state reached the summary model and reinjected summaries carried
  raw PII into model-bound context. Add a shared redact_text() seam: the
  compaction prompt is redacted in _build_summary_prompt (app_config
  already flows into the middleware) and DurableContextMiddleware redacts
  summary_text at reinjection via a new pii_redaction_config knob wired
  at both assembly sites.
- [P2] CUIT is 2+8+1 digits, not 2+10+1.
- [P2] Digit-anchored patterns use digit-aware lookarounds instead of
  Unicode \b, which CJK characters defeat (身份证110105… / 手机号138…).
- [P2] The international-phone pattern no longer treats newlines as
  separators, so a candidate cannot swallow the following numeric field
  and then fail validation as a whole.

* fix(pii): redact title input and reserve summary placeholders

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
xiaodu55 2026-09-19 11:26:46 +08:00 committed by GitHub
parent 058b2a49c5
commit 2ff006b0c0
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
12 changed files with 1263 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
README.md
View File

@ -1402,6 +1402,15 @@ The Web UI shows the active goal above the composer. The same command is availab
### Manual Context Compaction
Optional `pii_redaction.enabled` redacts detected identifiers in user messages,
remote tool results, compaction input, reinjected summaries, and configured
LLM title input. It is off by default. Existing summary placeholders reserve
indices so new values do not reuse them after compaction. No PII mapping is
persisted, so repeated values cannot be linked to a compacted source; numbering
may change when history or summary placeholders disappear. Raw thread text and
local fallback titles remain available for display; memory extraction is outside
this feature's scope.
The Web UI preserves persisted message order when merging history with live updates. Streaming steps around a persisted result inside the loaded history stay together, including steps that arrive after the result. Steps captured during compaction also remain visible before their persisted result when history has not refreshed and the UI has not rendered them yet.
Compaction keeps the current user request and summarizes older assistant/tool activity. When rescuing that request leaves an assistant/tool-only summary window, input trimming favors its most recent content. For mixed histories whose user-message anchor falls outside the trimming budget, compaction retains the existing final-message fallback. `summarization.trim_tokens_to_summarize` (4000 by default) controls trimming of the raw summary input; escaping and prompt formatting add overhead beyond that budget. Setting this option to `null` disables input trimming for the summary model; choose that only when the model can accept the full history being compacted.

1
backend/packages/harness/deerflow/agents/lead_agent/agent.py
View File

@ -608,6 +608,7 @@ def build_middlewares(
skills_container_path=resolved_app_config.skills.container_path,
skill_file_read_tool_names=resolved_app_config.summarization.skill_file_read_tool_names,
task_continuity_enabled=getattr(getattr(resolved_app_config, "task_continuity", None), "enabled", False) is True,
pii_redaction_config=getattr(resolved_app_config, "pii_redaction", None),
)
)

68
backend/packages/harness/deerflow/agents/middlewares/AGENTS.md
View File

@ -69,16 +69,18 @@ strict providers reject.
ordered by application — the last entry produced the final visible bytes — so an
observer classifies raw→visible transforms from facts rather than by sniffing
output wording.
4. **ThreadDataMiddleware** - Creates per-thread directories under the user's isolation scope (`backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); resolves identity via `resolve_runtime_user_id(runtime)`, including Gateway runtime context and standalone LangGraph Server auth, then falls back to the request ContextVar / `"default"`
5. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation (lead agent only); upload existence checks use the same runtime-resolved user bucket as thread-data creation
6. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state. The
4. **PiiRedactionMiddleware** - *(optional, `pii_redaction.enabled`, default off, #3190)* Rewrites PII in genuine user messages (`wrap_model_call`, request-scoped, raw text stays in thread state) and remote-content tool results (`wrap_tool_call`, ToolResultSanitizationMiddleware's allowlist incl. `Command.update.messages`) to irreversible placeholders (`[EMAIL_1]`, …). Deterministic regex detectors only, no new dependencies; national IDs run before the card detector so a checksum-valid resident ID whose digits also pass Luhn is never mislabeled `[CREDIT_CARD_n]`. One redactor per result keeps numbering continuous; existing summary/message tokens reserve indices before new values; no persistent mapping links compacted identities. Innermost Layer-1 wrapper, so budget-externalized copies hold redacted text; compaction, reinjected summaries, and title-model inputs are redacted; title fields are redacted before truncation. Memory extraction is a follow-up slice.
5. **ThreadDataMiddleware** - Creates per-thread directories under the user's isolation scope (`backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); resolves identity via `resolve_runtime_user_id(runtime)`, including Gateway runtime context and standalone LangGraph Server auth, then falls back to the request ContextVar / `"default"`
6. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation (lead agent only); upload existence checks use the same runtime-resolved user bucket as thread-data creation
7. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state. The
lead runtime normally owns the thread's physical Agent-skill projection;
delegated subagents and the prompt-only bootstrap agent are non-owners, so
their narrower discovery allowlists never rebuild the shared thread view or
force eager sandbox acquisition.
7. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., user interruption), preserving raw provider tool-call payloads in `additional_kwargs["tool_calls"]`; malformed tool-call names and arguments are sanitized in the model-bound request so strict OpenAI-compatible providers do not reject the next request
8. **LLMErrorHandlingMiddleware** - Converts provider/model failures to recoverable assistant errors. Normal completions without visible text or tool-call intent (including whitespace/reasoning-only responses) get at most one retry per run, then a marked visible fallback; empties never count toward the circuit breaker. Cancellation during admission, execution, retry events or backoff releases only this call's half-open probe (assigned under the circuit lock), then propagates unchanged without retry or failure accounting.
9. **Authorization / GuardrailMiddleware** - Up to two independent pre-tool-call gates run here. When `authorization.enabled`, the `AuthorizationProvider` instance already used for Layer 1 capability filtering is wrapped by `GuardrailAuthorizationAdapter` and reused for Layer 2 execution checks. A generated `tool_search` bypasses the adapter's second provider call only when the current build has a concrete deferred setup; its catalog was already filtered by Layer 1, and an ordinary same-named tool without that deferred setup receives no exemption. When `guardrails.enabled`, the explicitly configured `GuardrailProvider` is appended after authorization and still evaluates every call, including `tool_search`. Authorization therefore runs outermost and can deny before an external guardrail call; both use the existing middleware's fail-closed, audit, sync/async, and error-`ToolMessage` behavior. See the authorization RFC and [docs/GUARDRAILS.md](../../../../../docs/GUARDRAILS.md).
8. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., user interruption), preserving raw provider tool-call payloads in `additional_kwargs["tool_calls"]`; malformed tool-call names and arguments are sanitized in the model-bound request so strict OpenAI-compatible providers do not reject the next request
9. **LLMErrorHandlingMiddleware** - Converts provider/model failures to recoverable assistant errors. Async cancellation at admission, provider execution, retry events, or backoff releases only the call's own half-open probe (ownership assigned under the circuit lock), then propagates unchanged, without retry or failure accounting.
10. **Authorization / GuardrailMiddleware** - Up to two independent pre-tool-call gates run here. When `authorization.enabled`, the `AuthorizationProvider` instance already used for Layer 1 capability filtering is wrapped by `GuardrailAuthorizationAdapter` and reused for Layer 2 execution checks. A generated `tool_search` bypasses the adapter's second provider call only when the current build has a concrete deferred setup; its catalog was already filtered by Layer 1, and an ordinary same-named tool without that deferred setup receives no exemption. When `guardrails.enabled`, the explicitly configured `GuardrailProvider` is appended after authorization and still evaluates every call, including `tool_search`. Authorization therefore runs outermost and can deny before an external guardrail call; both use the existing middleware's fail-closed, audit, sync/async, and error-`ToolMessage` behavior. See the authorization RFC and [docs/GUARDRAILS.md](../../../../../docs/GUARDRAILS.md).
Every guardrail decision path publishes a neutral
`deerflow.authz.outcome.AuthorizationOutcome` into the per-run runtime context,
@ -86,10 +88,10 @@ strict providers reject.
`__authorization_outcome` key (so `build_run_config` strips caller-supplied
forgeries). Consumers pop it; the publisher and the consumer share only that
contract module.
10. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations before tool execution; command classification is **defense-in-depth and audit, not a security boundary** (the sandbox is the isolation boundary). Command substitution is judged by *position*, not the presence of `$(`: **command position** (`$(curl url)`, `` `curl url` ``, the word after `|`/`&&`/`;`, an `eval`/`source` argument) executes fetched content and is blocked; **value position** (`x=$(curl url)`, `echo $(curl url)`, an argument, a `for` word list) only captures output and passes (#4611). So `_HIGH_RISK_COMMAND_POSITION_PATTERNS` is matched anchored against each sub-command from `_split_compound_command(split_pipes=True)`, never the whole string; pipe-spanning rules (`| sh`, `base64 -d | ...`) still use `_classify_command`'s whole-command Pass 1. `_COMMAND_POSITION_PREFIX` extends the anchor over leading assignments and exec wrappers (`FOO=1 $(curl url)`, `env`/`command`/`builtin`/`exec`/`nohup`/`time`/`sudo`/`doas`); its assignment branch requires whitespace before the substitution, which keeps `x=$(curl url)` in value position. Two contexts are deliberately **position-blind** (matched whole-command in Pass 1, since they execute their input anywhere, e.g. `xargs sh -c "$(curl url)"`): an `eval`/`source` argument, and an interpreter **code-string flag** — `-c` (shells, `python`), `-e` (`perl`/`ruby`/`node`), `-p` (`perl`/`node`), `-r` (`php`) — plus the here-string (`<<<`) reaching the same place via stdin. All three substitution spellings (`$(`, `<(`, `` ` ``) share one `_RISKY_SUBSTITUTION` opener. An unquoted newline splits like `;` (else `echo hi\n$(curl url)` evades the anchored rules). Heredoc bodies are data: `_split_compound_command` records headers (`<<EOF`, `<<-EOF`, `<<'EOF'`) and consumes their bodies verbatim, so a body line starting `$(curl url)` isn't promoted to a command position; `<<<` (here-string, needs look-ahead + look-behind) and a `<<` inside `$(( ))`/`(( ))` (bit shift, arithmetic depth tracked with the quote flags) must not open one. This is a heuristic, not shell parsing — an unterminated body consumes the rest of the string, an unclosed `((` only disables heredoc detection, and the failure direction is always toward *more* command positions, not fewer. Known gaps: process substitution outside `eval`/`source` (`. <(curl u)`) is undetected, and two-step forms (`x=$(curl u); eval "$x"`) need dataflow analysis. No config gate — appended unconditionally in `_build_runtime_middlewares`, for both lead and subagents.
11. **ReadBeforeWriteMiddleware** - *(optional, `read_before_write.enabled`, default on)* Outermost write gate (#3857): `read_file` stamps a content hash on its ToolMessage; `write_file` (existing file, incl. append) and `str_replace` are blocked unless the newest mark for the path matches its current hash. Sits outside ToolProgress/ToolErrorHandling (a block consumes no ToolProgress slot); blocked results self-stamp `deerflow_tool_meta` and carry `deerflow_write_block` (`{path, tool}`). Marks live on messages, so summarization dropping the read invalidates the gate; writes never refresh marks. Gate check + execution are serialized per (thread, path); `"Error: ..."`-string sandboxes (AIO/E2B) fail open. It owns the composed call's sandbox authorization scope; `SandboxAuthorizationError` becomes an error ToolMessage. Its `wrap_model_call` swaps blocked calls' dead payload (`content`, `old_str`/`new_str`) for a deterministic placeholder in the model-bound request only (`elide_blocked_payloads`, `elide_min_chars`); state, receipts, and the journal keep the originals. Policy stays in the gate; the shared `tool_call_args` helper rewrites every arg surface together and every model-bound arg rewrite must use it.
12. **ToolProgressMiddleware** - *(optional, if `tool_progress.enabled`)* State-machine-based stagnation guard (RFC #3177). Outer wrapper around ToolErrorHandlingMiddleware so its tool wrapper receives results already stamped with `deerflow_tool_meta`. Recoverable problems stay WARNED with hints; retryable non-recoverable problems escalate to BLOCKED; stop-category failures block immediately. It is independent of LoopDetection's turn-level call-pattern guard. Effective phase changes emit `middleware:tool_progress`, including `reset` when a later agent invocation (for example, goal continuation) clears WARNED/BLOCKED state. The state machine supplies the action and count threshold (`null` for category rules, recovery, and reset); recorder calls happen after releasing the state lock. Events include the actual sync/async hook plus bounded status, error, and recovery fields. Server-owned recorder keys determine subagent attribution. Arguments, content, prompts, and hashes are never persisted; recorder failures are fail-open. See [event semantics](../../../../../docs/RUN_EVENT_STREAM.md).
13. **ToolReceiptMiddleware + ToolErrorHandlingMiddleware** - `ToolReceiptMiddleware` is *(optional, if `verification.receipts_enabled`, default on)*. It is the **outermost `wrap_tool_call` layer** — registered ahead of entries 9-12 — because Guardrail/SandboxAudit/ReadBeforeWrite/ToolProgress can short-circuit a call with their own ToolMessage (and SandboxAudit rebuilds medium-risk results); an inner receipt layer would silently gap the ledger on those results (ordering constraints in `deerflow.extensions.ordering`). Normal results still carry the `deerflow_tool_meta` status ToolErrorHandlingMiddleware stamps on the inner return path; short-circuit messages self-stamp meta or fall back to `message.status`. It stamps deterministic provenance (tool name, status, args/output hashes, byte count, timestamp) onto direct `ToolMessage` results and every matching `ToolMessage` carried in `Command.update.messages`, including delegated `task`, `present_file`, `view_image`, and `tool_search` results; before model calls it derives a hidden receipt ledger (display ids r1..rN) from message state, and when the 2,000-character budget is exceeded the newest receipts are retained in chronological order with their original ids plus an older-receipts omission marker. Rendering returns both the text and its retained receipt subset; every response that received a ledger carries only that exact server-owned subset, never omitted receipts. Snapshot validation accepts a strictly consecutive positive original-id range (for example `r24`–`r30`) rather than requiring `r1`, so subagent terminal citation verification resolves ids against evidence present in the citing turn even when later summarization drops and renumbers tool messages. Model-generated citation IDs are digit-bounded before integer conversion; oversized IDs are ignored as malformed input rather than raising through task write-back. Citation parsing deduplicates exact `(id, anchor)` pairs, not IDs alone, so repeated identical references stay compact while every distinct anchor claim is verified. Gateway strips delegated receipts/verdicts from external messages. `ToolErrorHandlingMiddleware` receives `AppConfig`, converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting, stamps every result with `deerflow_tool_meta` (status / error_type / recoverable_by_model / recommended_next_action / source) via `tool_result_meta.normalize_tool_result`, stamps structured metadata for task exception wrappers, and stamps skill-read metadata for downstream durable-context capture. Task tool result text is generated from the same status/result/error inputs as the structured metadata so callers do not hand-write a second protocol string.
11. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations before tool execution; command classification is **defense-in-depth and audit, not a security boundary** (the sandbox is the isolation boundary). Command substitution is judged by *position*, not the presence of `$(`: **command position** (`$(curl url)`, `` `curl url` ``, the word after `|`/`&&`/`;`, an `eval`/`source` argument) executes fetched content and is blocked; **value position** (`x=$(curl url)`, `echo $(curl url)`, an argument, a `for` word list) only captures output and passes (#4611). So `_HIGH_RISK_COMMAND_POSITION_PATTERNS` is matched anchored against each sub-command from `_split_compound_command(split_pipes=True)`, never the whole string; pipe-spanning rules (`| sh`, `base64 -d | ...`) still use `_classify_command`'s whole-command Pass 1. `_COMMAND_POSITION_PREFIX` extends the anchor over leading assignments and exec wrappers (`FOO=1 $(curl url)`, `env`/`command`/`builtin`/`exec`/`nohup`/`time`/`sudo`/`doas`); its assignment branch requires whitespace before the substitution, which keeps `x=$(curl url)` in value position. Two contexts are deliberately **position-blind** (matched whole-command in Pass 1, since they execute their input anywhere, e.g. `xargs sh -c "$(curl url)"`): an `eval`/`source` argument, and an interpreter **code-string flag** — `-c` (shells, `python`), `-e` (`perl`/`ruby`/`node`), `-p` (`perl`/`node`), `-r` (`php`) — plus the here-string (`<<<`) reaching the same place via stdin. All three substitution spellings (`$(`, `<(`, `` ` ``) share one `_RISKY_SUBSTITUTION` opener. An unquoted newline splits like `;` (else `echo hi\n$(curl url)` evades the anchored rules). Heredoc bodies are data: `_split_compound_command` records headers (`<<EOF`, `<<-EOF`, `<<'EOF'`) and consumes their bodies verbatim, so a body line starting `$(curl url)` isn't promoted to a command position; `<<<` (here-string, needs look-ahead + look-behind) and a `<<` inside `$(( ))`/`(( ))` (bit shift, arithmetic depth tracked with the quote flags) must not open one. This is a heuristic, not shell parsing — an unterminated body consumes the rest of the string, an unclosed `((` only disables heredoc detection, and the failure direction is always toward *more* command positions, not fewer. Known gaps: process substitution outside `eval`/`source` (`. <(curl u)`) is undetected, and two-step forms (`x=$(curl u); eval "$x"`) need dataflow analysis. No config gate — appended unconditionally in `_build_runtime_middlewares`, for both lead and subagents.
12. **ReadBeforeWriteMiddleware** - *(optional, `read_before_write.enabled`, default on)* Outermost write gate (#3857): `read_file` stamps a content hash on its ToolMessage; `write_file` (existing file, incl. append) and `str_replace` are blocked unless the newest mark for the path matches its current hash. Sits outside ToolProgress/ToolErrorHandling (a block consumes no ToolProgress slot); blocked results self-stamp `deerflow_tool_meta` and carry `deerflow_write_block` (`{path, tool}`). Marks live on messages, so summarization dropping the read invalidates the gate; writes never refresh marks. Gate check + execution are serialized per (thread, path); `"Error: ..."`-string sandboxes (AIO/E2B) fail open. It owns the composed call's sandbox authorization scope; `SandboxAuthorizationError` becomes an error ToolMessage. Its `wrap_model_call` swaps blocked calls' dead payload (`content`, `old_str`/`new_str`) for a deterministic placeholder in the model-bound request only (`elide_blocked_payloads`, `elide_min_chars`); state, receipts, and the journal keep the originals. Policy stays in the gate; the shared `tool_call_args` helper rewrites every arg surface together and every model-bound arg rewrite must use it.
13. **ToolProgressMiddleware** - *(optional, if `tool_progress.enabled`)* State-machine-based stagnation guard (RFC #3177). Outer wrapper around ToolErrorHandlingMiddleware so its tool wrapper receives results already stamped with `deerflow_tool_meta`. Recoverable problems stay WARNED with hints; retryable non-recoverable problems escalate to BLOCKED; stop-category failures block immediately. It is independent of LoopDetection's turn-level call-pattern guard. Effective phase changes emit `middleware:tool_progress`, including `reset` when a later agent invocation (for example, goal continuation) clears WARNED/BLOCKED state. The state machine supplies the action and count threshold (`null` for category rules, recovery, and reset); recorder calls happen after releasing the state lock. Events include the actual sync/async hook plus bounded status, error, and recovery fields. Server-owned recorder keys determine subagent attribution. Arguments, content, prompts, and hashes are never persisted; recorder failures are fail-open. See [event semantics](../../../../../docs/RUN_EVENT_STREAM.md).
14. **ToolReceiptMiddleware + ToolErrorHandlingMiddleware** - `ToolReceiptMiddleware` is *(optional, if `verification.receipts_enabled`, default on)*. It is the **outermost `wrap_tool_call` layer** — registered ahead of entries 10-13 — because Guardrail/SandboxAudit/ReadBeforeWrite/ToolProgress can short-circuit a call with their own ToolMessage (and SandboxAudit rebuilds medium-risk results); an inner receipt layer would silently gap the ledger on those results (ordering constraints in `deerflow.extensions.ordering`). Normal results still carry the `deerflow_tool_meta` status ToolErrorHandlingMiddleware stamps on the inner return path; short-circuit messages self-stamp meta or fall back to `message.status`. It stamps deterministic provenance (tool name, status, args/output hashes, byte count, timestamp) onto direct `ToolMessage` results and every matching `ToolMessage` carried in `Command.update.messages`, including delegated `task`, `present_file`, `view_image`, and `tool_search` results; before model calls it derives a hidden receipt ledger (display ids r1..rN) from message state, and when the 2,000-character budget is exceeded the newest receipts are retained in chronological order with their original ids plus an older-receipts omission marker. Rendering returns both the text and its retained receipt subset; every response that received a ledger carries only that exact server-owned subset, never omitted receipts. Snapshot validation accepts a strictly consecutive positive original-id range (for example `r24`–`r30`) rather than requiring `r1`, so subagent terminal citation verification resolves ids against evidence present in the citing turn even when later summarization drops and renumbers tool messages. Model-generated citation IDs are digit-bounded before integer conversion; oversized IDs are ignored as malformed input rather than raising through task write-back. Citation parsing deduplicates exact `(id, anchor)` pairs, not IDs alone, so repeated identical references stay compact while every distinct anchor claim is verified. Gateway strips delegated receipts/verdicts from external messages. `ToolErrorHandlingMiddleware` receives `AppConfig`, converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting, stamps every result with `deerflow_tool_meta` (status / error_type / recoverable_by_model / recommended_next_action / source) via `tool_result_meta.normalize_tool_result`, stamps structured metadata for task exception wrappers, and stamps skill-read metadata for downstream durable-context capture. Task tool result text is generated from the same status/result/error inputs as the structured metadata so callers do not hand-write a second protocol string.
Authorization identity is independent of enforcement. Gateway strips client identity overrides: only the server auth source sets `is_internal`, and only authenticated IM `body.context` supplies `channel_user_id` (never `body.config`). `build_principal_from_context` applies role defaults, strict provenance, and copied attributes; RBAC rejects unknown defaults. Delegation and `GuardrailMiddleware` share this identity. Layer 1 precedes deferred assembly across agent paths and its provider is reused for Layer 2; framework skill/memory ordering stays stable. Trusted `DeerFlowClient.stream()` accepts identity overrides. Its graph key always includes effective storage `user_id` and, when enforced, the full Principal; nested attributes are copied so mutation cannot hide stale cache state.
@ -103,31 +105,31 @@ Before changing a later authorization phase, read the [authorization RFC](../../
**Lead-only middlewares** (`build_middlewares`, appended after the base):
14. **DynamicContextMiddleware** - Injects date and optional memory outside the static prompt. Opt-out removes its frozen server memory but retains date/user messages. Date follows server-local time unless `DEER_FLOW_DATE_TIMEZONE` names an IANA zone (invalid values fall back locally).
15. **SkillActivationMiddleware** - Detects strict `/skill-name task` syntax on the latest real user message, resolves only enabled and runtime-allowed skills, injects the `SKILL.md` body as hidden current-turn context, and records a `middleware:skill_activation` audit event
16. **SkillToolPolicyMiddleware** - Applies `allowed-tools` only after real activation; passive enabled skills and a custom agent's configured skill allowlist do not clamp the lead toolset. A run-scoped slash activation is authoritative and suppresses `skill_context` as a policy source, so reading another skill cannot widen the explicit skill's tools; without slash activation, skills captured after configured `read_file` loads retain the existing union semantics. The middleware filters model-visible schemas and blocks unauthorized execution, resolving canonical paths against the live enabled/agent-allowed registry on every model call, then stores a versioned, JSON-safe, middleware-token-bound decision signed by policy source plus active paths in run context for the resulting tool calls to reuse. The next model call always refreshes it, and malformed, foreign, stale, or unmatched decisions fall back to live resolution. `tool_search` and `describe_skill` remain framework-safe discovery tools under a restrictive policy; they may reveal or promote metadata, but a deferred business tool must still be declared by the active policy before its schema or execution can survive the policy middleware. The decision's owner token is authorization-sensitive, so its reserved context key is owned by `runtime.secret_context` and included in `REDACTED_CONTEXT_KEYS` for observable and persisted context copies. Registry load failures and a non-empty active set with no authorized skill fail closed to framework-safe tools; an individual stale path is skipped only when at least one valid active skill remains. This is best-effort behavioral scoping rather than a hard security boundary: alternate loads such as `bash cat` are not captured, and bounded autonomous `skill_context` can evict old entries. `task` is not framework-exempt, so a restricted skill cannot delegate around its policy. The middleware must remain immediately after `SkillActivationMiddleware` (which publishes the slash source through `runtime.secret_context`'s public path helpers authenticated by a required token shared only within the assembled middleware chain) and immediately before `DurableContextMiddleware`; assembly and compiled-graph tests pin ordering, token sharing, schema filtering, and execution blocking.
17. **DurableContextMiddleware** - Captures `task` delegations into `ThreadState.delegations` (including in-progress dispatches and terminal result summaries) and loaded skill-file references (name/path/description, parsed in-memory - not the body) into `ThreadState.skill_context` before summarization can compact the paired tool-call/result messages, then projects durable context into each model request. Static authority rules are injected as a `SystemMessage`; untrusted field values (`summary_text`, delegation results, skill descriptions) are injected separately as a hidden `HumanMessage` data block so compressed history, delegated work, and which skills are active stay visible without being stored as `messages` or promoted to system-role instructions. `build_subagent_runtime_middlewares` also attaches this middleware immediately before subagent summarization so a compacted `summary_text` is projected ahead of a preserved assistant/tool tail instead of leaving strict providers with an assistant-first request.
18. **SummarizationMiddleware** - *(optional, if enabled)* Compacts near token limits; memory flush follows runtime policy, while manual compaction trusts the checkpoint-bound agent, so opt-out never captures removed turns. It preserves the latest real user request by ID and tagged DynamicContext reminders while allowing stale ID-swap peers into the summary. Moving the cutoff backward can retain old AI/tool turns and make first-turn compaction a no-op.
19. **TodoListMiddleware** - *(optional, if `is_plan_mode`)* Task tracking with the `write_todos` tool
20. **TokenUsageMiddleware** - *(optional, if `token_usage.enabled`)* Records token usage metrics; subagent usage is read from terminal `ToolMessage.additional_kwargs` in the current run and merged back into the dispatching AIMessage by message position. The same state update marks the ToolMessage with `subagent_token_usage_attributed=true`, so checkpoint replay or middleware re-entry cannot add the cumulative snapshot twice; missing/malformed usage or a result with no matching dispatch remains unmarked and retryable.
21. **TitleMiddleware** - Auto-generates the thread title after the first complete exchange and normalizes structured message content before prompting the title model. If a first-turn run is interrupted before this middleware can write a title, `runtime/runs/worker.py` keeps the run in a finalizing state, persists a local fallback title from the latest checkpoint or original run input, and then syncs it to `threads_meta.display_name`. Replacement runs admitted by `multitask_strategy="interrupt"` / `"rollback"` wait for older same-thread finalization before entering the graph; the interrupted run only skips the fallback title write once a later run has started and may have advanced the checkpoint.
22. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses); captures the runtime-resolved user so standalone LangGraph Server reads and writes stay in the same bucket
23. **ViewImageMiddleware** - *(optional, if the model supports vision)* Appends a hidden HumanMessage with base64 image data, identified by a reserved ID prefix plus a server-owned metadata marker, to `ModelRequest.messages` in `wrap_model_call` / `awrap_model_call`. The payload lives only in that request and is never returned as a state update, so no checkpoint carries it and an interrupted run cannot strand it in history; state keeps only the lightweight `viewed_images` metadata. It owns that context and rebuilds it per call: its own message is swept out of the request first — a thread checkpointed by the earlier `before_model`/`after_model` pair (which wrote the payload into state and took it back out with `RemoveMessage`) can carry one that reached state but was never removed, and leaving it in would resend that base64 in every later request for the life of the thread — then a freshly built one is appended when warranted. The sweep requires both the reserved ID prefix and the server-owned marker, so a client cannot get its own message dropped; unmarked leftovers predating the marker are left in place and merely not duplicated
24. **McpRoutingMiddleware** - Auto-promotes deferred schemas matching the latest real user message before `DeferredToolFilterMiddleware`, without executing tools. New names emit `middleware:tool_promotion` with `source=routing_hint`; repeat passes emit nothing
25. **DeferredToolPromotionAuditMiddleware** - Observes final `tool_search` `Command`s; keep it outer of `SkillToolPolicyMiddleware` so denied names are excluded. It atomically claims new names per lead run or subagent execution to dedupe parallel searches, derives subagent attribution from the server-installed recorder, returns the original `Command`, and omits private payloads
26. **DeferredToolFilterMiddleware** - *(optional, if `tool_search.enabled`)* Hides deferred (MCP) tool schemas from the bound model until `tool_search` or `McpRoutingMiddleware` promotes them (reads per-thread promotions from `ThreadState.promoted`, hash-scoped)
27. **SystemMessageCoalescingMiddleware** - Merges every SystemMessage into a single leading SystemMessage per request; provider-agnostic fix for strict backends (vLLM/SGLang/Qwen/Anthropic) that reject non-leading system messages. Touches the per-request payload only (checkpoint state unchanged); on midnight crossings only the latest `dynamic_context_reminder` SystemMessage survives. The subagent builder places its date-only context middleware immediately before this coalescer, so the built-in subagent prompt and hidden date reminder still reach providers as one leading system block
28. **SubagentLimitMiddleware** - *(optional, if `subagent_enabled`)* Truncates excess ordinary `task` tool calls to enforce both the per-response concurrency limit (`max_concurrent_subagents`, resolved against startup `subagent_runtime.max_running` and the 1-64 safety range before construction) and the per-run total delegation cap (`max_total_subagents` runtime override or `subagents.max_total_per_run`, default 6, clamped to 1-50). The total cap counts current-run entries in the durable delegation ledger (entries are tagged with `run_id` when captured), so repeated planning checkpoints in one run cannot keep launching legal-sized batches indefinitely, while later user turns in the same thread get a fresh run budget. Explicit durable `batch_task` calls are a separate mode with persisted total/live/running limits and are not rewritten into ordinary ledger entries. If the ordinary cap is exhausted, the middleware strips remaining `task` calls, forces `finish_reason="stop"`, and appends a visible limit note so the run can synthesize existing results instead of ending with an empty tool-call response.
29. **LoopDetectionMiddleware** - *(optional, if `loop_detection.enabled`)* Detects repeated tool-call loops; hard-stop clears structured, raw, and content-block tool calls before forcing a final text answer; stamps `loop_capped` via `consume_stop_reason` (#3875 Phase 2), symmetric to `TokenBudgetMiddleware`; persists warned-state transitions (first per call hash or per tool-frequency burst) and hard stops as `middleware:loop_detection`, attributed with `is_subagent` and the optional `agent_id`, without tool arguments, message content, tool results, or argument-derived hashes. Ordinary task subagents get dedicated recorder keys through a parent-loop proxy; never pass `RunJournal` into their isolated loop. Durable batch subagents have no parent run journal and do not persist these transitions
15. **DynamicContextMiddleware** - Injects date and optional memory outside the static prompt. Opt-out removes its frozen server memory but retains date/user messages. Date follows server-local time unless `DEER_FLOW_DATE_TIMEZONE` names an IANA zone (invalid values fall back locally).
16. **SkillActivationMiddleware** - Detects strict `/skill-name task` syntax on the latest real user message, resolves only enabled and runtime-allowed skills, injects the `SKILL.md` body as hidden current-turn context, and records a `middleware:skill_activation` audit event
17. **SkillToolPolicyMiddleware** - Applies `allowed-tools` only after real activation; passive enabled skills and a custom agent's configured skill allowlist do not clamp the lead toolset. A run-scoped slash activation is authoritative and suppresses `skill_context` as a policy source, so reading another skill cannot widen the explicit skill's tools; without slash activation, skills captured after configured `read_file` loads retain the existing union semantics. The middleware filters model-visible schemas and blocks unauthorized execution, resolving canonical paths against the live enabled/agent-allowed registry on every model call, then stores a versioned, JSON-safe, middleware-token-bound decision signed by policy source plus active paths in run context for the resulting tool calls to reuse. The next model call always refreshes it, and malformed, foreign, stale, or unmatched decisions fall back to live resolution. `tool_search` and `describe_skill` remain framework-safe discovery tools under a restrictive policy; they may reveal or promote metadata, but a deferred business tool must still be declared by the active policy before its schema or execution can survive the policy middleware. The decision's owner token is authorization-sensitive, so its reserved context key is owned by `runtime.secret_context` and included in `REDACTED_CONTEXT_KEYS` for observable and persisted context copies. Registry load failures and a non-empty active set with no authorized skill fail closed to framework-safe tools; an individual stale path is skipped only when at least one valid active skill remains. This is best-effort behavioral scoping rather than a hard security boundary: alternate loads such as `bash cat` are not captured, and bounded autonomous `skill_context` can evict old entries. `task` is not framework-exempt, so a restricted skill cannot delegate around its policy. The middleware must remain immediately after `SkillActivationMiddleware` (which publishes the slash source through `runtime.secret_context`'s public path helpers authenticated by a required token shared only within the assembled middleware chain) and immediately before `DurableContextMiddleware`; assembly and compiled-graph tests pin ordering, token sharing, schema filtering, and execution blocking.
18. **DurableContextMiddleware** - Captures `task` delegations into `ThreadState.delegations` (including in-progress dispatches and terminal result summaries) and loaded skill-file references (name/path/description, parsed in-memory - not the body) into `ThreadState.skill_context` before summarization can compact the paired tool-call/result messages, then projects durable context into each model request. Static authority rules are injected as a `SystemMessage`; untrusted field values (`summary_text`, delegation results, skill descriptions) are injected separately as a hidden `HumanMessage` data block so compressed history, delegated work, and which skills are active stay visible without being stored as `messages` or promoted to system-role instructions. `build_subagent_runtime_middlewares` also attaches this middleware immediately before subagent summarization so a compacted `summary_text` is projected ahead of a preserved assistant/tool tail instead of leaving strict providers with an assistant-first request.
19. **SummarizationMiddleware** - *(optional, if enabled)* Compacts near token limits; memory flush follows runtime policy, while manual compaction trusts the checkpoint-bound agent, so opt-out never captures removed turns. It preserves the latest real user request by ID and tagged DynamicContext reminders while allowing stale ID-swap peers into the summary. Moving the cutoff backward can retain old AI/tool turns and make first-turn compaction a no-op.
20. **TodoListMiddleware** - *(optional, if `is_plan_mode`)* Task tracking with the `write_todos` tool
21. **TokenUsageMiddleware** - *(optional, if `token_usage.enabled`)* Records token usage metrics; subagent usage is read from terminal `ToolMessage.additional_kwargs` in the current run and merged back into the dispatching AIMessage by message position. The same state update marks the ToolMessage with `subagent_token_usage_attributed=true`, so checkpoint replay or middleware re-entry cannot add the cumulative snapshot twice; missing/malformed usage or a result with no matching dispatch remains unmarked and retryable.
22. **TitleMiddleware** - Auto-generates the thread title after the first complete exchange and normalizes structured message content before prompting the title model. If a first-turn run is interrupted before this middleware can write a title, `runtime/runs/worker.py` keeps the run in a finalizing state, persists a local fallback title from the latest checkpoint or original run input, and then syncs it to `threads_meta.display_name`. Replacement runs admitted by `multitask_strategy="interrupt"` / `"rollback"` wait for older same-thread finalization before entering the graph; the interrupted run only skips the fallback title write once a later run has started and may have advanced the checkpoint.
23. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses); captures the runtime-resolved user so standalone LangGraph Server reads and writes stay in the same bucket
24. **ViewImageMiddleware** - *(optional, if the model supports vision)* Appends a hidden HumanMessage with base64 image data, identified by a reserved ID prefix plus a server-owned metadata marker, to `ModelRequest.messages` in `wrap_model_call` / `awrap_model_call`. The payload lives only in that request and is never returned as a state update, so no checkpoint carries it and an interrupted run cannot strand it in history; state keeps only the lightweight `viewed_images` metadata. It owns that context and rebuilds it per call: its own message is swept out of the request first — a thread checkpointed by the earlier `before_model`/`after_model` pair (which wrote the payload into state and took it back out with `RemoveMessage`) can carry one that reached state but was never removed, and leaving it in would resend that base64 in every later request for the life of the thread — then a freshly built one is appended when warranted. The sweep requires both the reserved ID prefix and the server-owned marker, so a client cannot get its own message dropped; unmarked leftovers predating the marker are left in place and merely not duplicated
25. **McpRoutingMiddleware** - Auto-promotes deferred schemas matching the latest real user message before `DeferredToolFilterMiddleware`, without executing tools. New names emit `middleware:tool_promotion` with `source=routing_hint`; repeat passes emit nothing
26. **DeferredToolPromotionAuditMiddleware** - Observes final `tool_search` `Command`s; keep it outer of `SkillToolPolicyMiddleware` so denied names are excluded. It atomically claims new names per lead run or subagent execution to dedupe parallel searches, derives subagent attribution from the server-installed recorder, returns the original `Command`, and omits private payloads
27. **DeferredToolFilterMiddleware** - *(optional, if `tool_search.enabled`)* Hides deferred (MCP) tool schemas from the bound model until `tool_search` or `McpRoutingMiddleware` promotes them (reads per-thread promotions from `ThreadState.promoted`, hash-scoped)
28. **SystemMessageCoalescingMiddleware** - Merges every SystemMessage into a single leading SystemMessage per request; provider-agnostic fix for strict backends (vLLM/SGLang/Qwen/Anthropic) that reject non-leading system messages. Touches the per-request payload only (checkpoint state unchanged); on midnight crossings only the latest `dynamic_context_reminder` SystemMessage survives. The subagent builder places its date-only context middleware immediately before this coalescer, so the built-in subagent prompt and hidden date reminder still reach providers as one leading system block
29. **SubagentLimitMiddleware** - *(optional, if `subagent_enabled`)* Truncates excess ordinary `task` tool calls to enforce both the per-response concurrency limit (`max_concurrent_subagents`, resolved against startup `subagent_runtime.max_running` and the 1-64 safety range before construction) and the per-run total delegation cap (`max_total_subagents` runtime override or `subagents.max_total_per_run`, default 6, clamped to 1-50). The total cap counts current-run entries in the durable delegation ledger (entries are tagged with `run_id` when captured), so repeated planning checkpoints in one run cannot keep launching legal-sized batches indefinitely, while later user turns in the same thread get a fresh run budget. Explicit durable `batch_task` calls are a separate mode with persisted total/live/running limits and are not rewritten into ordinary ledger entries. If the ordinary cap is exhausted, the middleware strips remaining `task` calls, forces `finish_reason="stop"`, and appends a visible limit note so the run can synthesize existing results instead of ending with an empty tool-call response.
30. **LoopDetectionMiddleware** - *(optional, if `loop_detection.enabled`)* Detects repeated tool-call loops; hard-stop clears structured, raw, and content-block tool calls before forcing a final text answer; stamps `loop_capped` via `consume_stop_reason` (#3875 Phase 2), symmetric to `TokenBudgetMiddleware`; persists warned-state transitions (first per call hash or per tool-frequency burst) and hard stops as `middleware:loop_detection`, attributed with `is_subagent` and the optional `agent_id`, without tool arguments, message content, tool results, or argument-derived hashes. Ordinary task subagents get dedicated recorder keys through a parent-loop proxy; never pass `RunJournal` into their isolated loop. Durable batch subagents have no parent run journal and do not persist these transitions
State is run-scoped: new user runs get fresh budgets; same-run goal
continuations share history. Keep sibling warnings isolated and lifecycle
hooks topology-stable. Before changing this guard, read
[Loop detection lifecycle](../../../../../docs/LOOP_DETECTION.md) for
fallback identity, cleanup/LRU/reset, severity ordering, and test invariants.
30. **TokenBudgetMiddleware** - `token_budget.enabled`: shares run-ID budgets across continuations; missing/invalid IDs clear invocation state.
31. **Custom middlewares** - *(optional)* Any `custom_middlewares` passed to `build_middlewares` are injected here, before config-declared extensions and the terminal-response/safety/clarification tail
32. **Configured extension middlewares** - `extensions.middlewares` in `config.yaml` or `extensions_config.json` optionally accepts `module.path:ClassName` strings or `{class, kwargs}` objects. `deerflow.reflection.resolve_class` loads `AgentMiddleware` classes; import, class, and constructor errors fail agent creation. `kwargs` must be JSON-compatible; YAML dates/timestamps become ISO strings. Order: built-ins/custom and loop/token guards → extensions → terminal-response/safety/clarification tail. Subagents share the list before their safety tail; separate lead/subagent lists are unsupported. Trusted operator config only: paths instantiate arbitrary code. Gateway skill/MCP toggles preserve it in raw JSON; adding an API write path requires explicit trust-boundary review.
33. **TerminalResponseMiddleware** - After tools following the latest real user message, an assistant response without visible text or tool intent gets a marked visible fallback in the same step. Preserves content blocks; no graph retry or separate length-reason vocabulary.
34. **ModelLengthFinishReasonMiddleware** - A length-detector match stamps `stop_reason=model_length_capped` and ends the tool loop. Suppresses all structured/raw calls and native `tool_use` blocks, even fully parsed calls. Keeps text/thinking blocks; appends a length notice if no visible text exists. Audit metadata keeps detector, reason, call count/names, never suppressed arguments.
35. **SafetyFinishReasonMiddleware** - *(optional, if `safety_finish_reason.enabled`)* Suppresses tool execution when the provider safety-terminated the response (e.g. `finish_reason=content_filter`); registered after terminal-response/custom/configured middlewares so LangChain's reverse-order `after_model` dispatch runs it first
36. **ClarificationMiddleware** - Intercepts `ask_clarification`, writes a readable `ToolMessage.content` fallback plus a structured `ToolMessage.artifact.human_input` payload, and interrupts via `Command(goto=END)` (must be last). `after_model` drops same-turn sibling tool calls so they cannot run before the user answers; a malformed `ask_clarification` parked on `invalid_tool_calls` is the same stop signal. `disable_clarification` runs keep the siblings. Payloads are versioned — legacy `free_text`/`choice_with_other` stay `version: 1`; the v2 `form` mode (from `fields`) is `version: 2` so older frontends reject it and fall back to plain text. Field normalization is deterministic and lives in the middleware (it short-circuits before tool execution, so tool-arg typing gives no runtime validation), and it is atomic: any structurally broken entry — non-dict, bad/duplicate name, a name colliding with a JS `Object.prototype` member (`__proto__`/`constructor`), or exceeding the caps (16 fields / 24 options per field / 200 chars per text / `MAX_FORM_SERIALIZED_BYTES` = 16KB UTF-8, the per-item caps alone admitting forms whose IM text fallback overruns channel limits) — degrades the whole form to the legacy option/free-text modes, so a card never renders "complete" while missing a field. Benign issues degrade locally (unknown types — incl. unhashable JSON like `type: []`, which must not raise from the membership probe — and option-less selects become `text`); options are trimmed/deduped with blanks dropped (form- and top-level) since the frontend rejects blank labels. XML-to-dict option payloads are recursively flattened from dict/list containers in source order, scalar leaves kept, residual XML tags stripped before that trimming. Checkboxes are booleans defaulting to "no"; `required` on one means consent semantics. The response protocol is unchanged (v1 `text`/`option`): form cards submit a text summary as `response_kind: "text"`, so journal persistence needs no new allowlist entries. Because this middleware can short-circuit before `on_tool_end`, `RunJournal` does a root-run reconciliation for `ToolMessage`s whose `tool_call_id` came from the current run, so cards survive checkpoint compaction. That reconciliation is **not** `ask_clarification`-only — any middleware that answers a tool call has the same gap, and a result the user saw must not vanish on reload (#4666 — `ReadBeforeWriteMiddleware` blocked-write errors reached the UI but not the event store). It is bounded by three conditions, not a name allowlist: the message is user-visible, the call belongs to this run's **lead agent** (`_remember_current_run_tool_calls` records lead-agent calls only; subagent results stay in `subagent.step`), and it is not already persisted. Human Input Card replies are `hide_from_ui` `HumanMessage`s with `additional_kwargs.human_input_response`; `RunJournal` persists only allowlisted hidden sources (currently `ask_clarification`) as `llm.human.input`.
31. **TokenBudgetMiddleware** - `token_budget.enabled`: shares run-ID budgets across continuations; missing/invalid IDs clear invocation state.
32. **Custom middlewares** - *(optional)* Any `custom_middlewares` passed to `build_middlewares` are injected here, before config-declared extensions and the terminal-response/safety/clarification tail
33. **Configured extension middlewares** - `extensions.middlewares` in `config.yaml` or `extensions_config.json` optionally accepts `module.path:ClassName` strings or `{class, kwargs}` objects. `deerflow.reflection.resolve_class` loads `AgentMiddleware` classes; import, class, and constructor errors fail agent creation. `kwargs` must be JSON-compatible; YAML dates/timestamps become ISO strings. Order: built-ins/custom and loop/token guards → extensions → terminal-response/safety/clarification tail. Subagents share the list before their safety tail; separate lead/subagent lists are unsupported. Trusted operator config only: paths instantiate arbitrary code. Gateway skill/MCP toggles preserve it in raw JSON; adding an API write path requires explicit trust-boundary review.
34. **TerminalResponseMiddleware** - When a provider returns an empty terminal `AIMessage` after tool execution, injects a hidden recovery prompt and retries the model once; a second empty response is replaced in checkpoint state by a visible error fallback marked for the run worker, so the run finishes as an error instead of a silent success
35. **ModelLengthFinishReasonMiddleware** - Records `stop_reason=model_length_capped` when provider-specific length detectors match a terminal `AIMessage` without tool-call intent (`finish_reason=length` / `MAX_TOKENS`, or `stop_reason=max_tokens`), preserving the original assistant content and never reparsing textual tool-call-like envelopes
36. **SafetyFinishReasonMiddleware** - *(optional, if `safety_finish_reason.enabled`)* Suppresses tool execution when the provider safety-terminated the response (e.g. `finish_reason=content_filter`); registered after terminal-response/custom/configured middlewares so LangChain's reverse-order `after_model` dispatch runs it first
37. **ClarificationMiddleware** - Intercepts `ask_clarification`, writes a readable `ToolMessage.content` fallback plus a structured `ToolMessage.artifact.human_input` payload, and interrupts via `Command(goto=END)` (must be last). `after_model` drops same-turn sibling tool calls so they cannot run before the user answers; a malformed `ask_clarification` parked on `invalid_tool_calls` is the same stop signal. `disable_clarification` runs keep the siblings. Payloads are versioned — legacy `free_text`/`choice_with_other` stay `version: 1`; the v2 `form` mode (from `fields`) is `version: 2` so older frontends reject it and fall back to plain text. Field normalization is deterministic and lives in the middleware (it short-circuits before tool execution, so tool-arg typing gives no runtime validation), and it is atomic: any structurally broken entry — non-dict, bad/duplicate name, a name colliding with a JS `Object.prototype` member (`__proto__`/`constructor`), or exceeding the caps (16 fields / 24 options per field / 200 chars per text / `MAX_FORM_SERIALIZED_BYTES` = 16KB UTF-8, the per-item caps alone admitting forms whose IM text fallback overruns channel limits) — degrades the whole form to the legacy option/free-text modes, so a card never renders "complete" while missing a field. Benign issues degrade locally (unknown types — incl. unhashable JSON like `type: []`, which must not raise from the membership probe — and option-less selects become `text`); options are trimmed/deduped with blanks dropped (form- and top-level) since the frontend rejects blank labels. XML-to-dict option payloads are recursively flattened from dict/list containers in source order, scalar leaves kept, residual XML tags stripped before that trimming. Checkboxes are booleans defaulting to "no"; `required` on one means consent semantics. The response protocol is unchanged (v1 `text`/`option`): form cards submit a text summary as `response_kind: "text"`, so journal persistence needs no new allowlist entries. Because this middleware can short-circuit before `on_tool_end`, `RunJournal` does a root-run reconciliation for `ToolMessage`s whose `tool_call_id` came from the current run, so cards survive checkpoint compaction. That reconciliation is **not** `ask_clarification`-only — any middleware that answers a tool call has the same gap, and a result the user saw must not vanish on reload (#4666 — `ReadBeforeWriteMiddleware` blocked-write errors reached the UI but not the event store). It is bounded by three conditions, not a name allowlist: the message is user-visible, the call belongs to this run's **lead agent** (`_remember_current_run_tool_calls` records lead-agent calls only; subagent results stay in `subagent.step`), and it is not already persisted. Human Input Card replies are `hide_from_ui` `HumanMessage`s with `additional_kwargs.human_input_response`; `RunJournal` persists only allowlisted hidden sources (currently `ask_clarification`) as `llm.human.input`.

7
backend/packages/harness/deerflow/agents/middlewares/durable_context_middleware.py
View File

@ -24,9 +24,11 @@ from langgraph.runtime import Runtime
from deerflow.agents.middlewares.delegation_ledger import extract_delegations, render_delegation_ledger
from deerflow.agents.middlewares.message_utils import insert_after_leading_system_messages
from deerflow.agents.middlewares.pii_redaction_middleware import redact_text
from deerflow.agents.middlewares.skill_context import extract_skills, render_skill_context
from deerflow.agents.task_continuity.state import normalize_task_history, normalize_task_notes
from deerflow.agents.thread_state import _DELEGATION_LEDGER_MAX_ENTRIES, TERMINAL_STATUSES
from deerflow.config.pii_redaction_config import PiiRedactionConfig
from deerflow.config.summarization_config import DEFAULT_SKILL_FILE_READ_TOOL_NAMES
from deerflow.constants import DEFAULT_SKILLS_CONTAINER_PATH
from deerflow.runtime.context_keys import CURRENT_RUN_PRE_EXISTING_MESSAGE_IDS_KEY
@ -234,9 +236,11 @@ class DurableContextMiddleware(AgentMiddleware[AgentState]):
skills_container_path: str | None = None,
skill_file_read_tool_names: Collection[str] | None = None,
task_continuity_enabled: bool = False,
pii_redaction_config: PiiRedactionConfig | None = None,
) -> None:
super().__init__()
self._task_continuity_enabled = task_continuity_enabled
self._pii_redaction_config = pii_redaction_config
self._skills_root = _normalize_skills_root(skills_container_path)
self._skill_read_tool_names = frozenset(DEFAULT_SKILL_FILE_READ_TOOL_NAMES if skill_file_read_tool_names is None else skill_file_read_tool_names)
@ -246,6 +250,7 @@ class DurableContextMiddleware(AgentMiddleware[AgentState]):
"skills_container_path": self._skills_root,
"skill_file_read_tool_names": sorted(self._skill_read_tool_names),
"task_continuity_enabled": self._task_continuity_enabled,
"pii_redaction_enabled": bool(self._pii_redaction_config and self._pii_redaction_config.enabled),
}
@override
@ -293,7 +298,7 @@ class DurableContextMiddleware(AgentMiddleware[AgentState]):
def _inject(self, request: ModelRequest) -> ModelRequest:
state = request.state or {}
data_block = _render_durable_context_data(
state.get("summary_text"),
redact_text(state.get("summary_text"), self._pii_redaction_config),
state.get("delegations") or [],
state.get("skill_context") or [],
(state.get("task_notes") or {}) if self._task_continuity_enabled else None,

463
backend/packages/harness/deerflow/agents/middlewares/pii_redaction_middleware.py Normal file
View File

@ -0,0 +1,463 @@
"""PII redaction middleware for model-bound context (issue #3190).
Detects personally identifiable information in the two untrusted-content entry
points — genuine user messages and remote-content tool results — and rewrites
it to irreversible placeholders (``[EMAIL_1]`` …) before it reaches the model.
Complements the structural guardrails: ``InputSanitizationMiddleware``
neutralizes injection tags in user input and ``ToolResultSanitizationMiddleware``
does the same for remote tool results; neither inspects *content* for PII.
v1 is deterministic-only: fixed regex detectors with checksum validation where
the identifier format defines one (Luhn for card numbers, mod-11 for CN resident
IDs and CPF), no model calls, no new dependencies. Redaction is irreversible —
no mapping table is stored, so there is nothing to protect and no
re-identification path.
Scope model (mirrors the structural guardrails):
* the user-message rewrite is request-scoped — thread state keeps the raw text,
so the UI still shows the original message and the whole conversation is
re-redacted on every model call. Existing summary/message placeholders reserve
their indices before new values are assigned, avoiding collisions after
compaction. Without a persisted mapping, a repeated raw value cannot be
linked to a placeholder whose source was compacted away;
* tool-result redaction runs at the tool boundary (``wrap_tool_call``) with the
same allowlist as ``ToolResultSanitizationMiddleware`` (first-party web tools
by name, MCP tools via their ``deerflow_mcp`` tag), so redacted text is what
enters model context in the first place; placeholders restart per result;
* subagents are covered because ``build_subagent_runtime_middlewares`` reuses
this base;
* NOT covered in v1: the memory-extraction path (follow-up slice per the issue
discussion). Allowlisted tool results are redacted before budget externalization.
* The compaction and durable-context seams run outside ``wrap_model_call``;
:func:`redact_text` is the shared entry point they call, wired from
SummarizationMiddleware (compaction input) and DurableContextMiddleware
(reinjected ``summary_text``); TitleMiddleware redacts its complete user and
assistant fields before truncation and direct model invocation.
Detector order is fixed and pinned by a regression test; email → api_key →
national_id → credit_card → phone. Checksum-gated national IDs run *before*
the credit-card detector so an 18-digit resident ID whose digit run also
passes Luhn is never consumed as a card; unambiguous prefix/format patterns
(email, API keys) rewrite first, and phones last see only digits the stronger
gates did not claim.
"""
from __future__ import annotations
import logging
import re
from collections.abc import Awaitable, Callable, Sequence
from dataclasses import dataclass
from dataclasses import replace as dc_replace
from typing import override
from langchain.agents import AgentState
from langchain.agents.middleware import AgentMiddleware
from langchain.agents.middleware.types import (
ModelCallResult,
ModelRequest,
ModelResponse,
)
from langchain_core.messages import HumanMessage, ToolMessage
from langgraph.errors import GraphBubbleUp
from langgraph.prebuilt.tool_node import ToolCallRequest
from langgraph.types import Command
from deerflow.agents.middlewares.message_utils import requires_input_sanitization
from deerflow.agents.middlewares.tool_result_sanitization_middleware import _REMOTE_CONTENT_TOOL_NAMES
from deerflow.agents.middlewares.tool_transform_meta import append_tool_transform
from deerflow.config.pii_redaction_config import PiiRedactionConfig
from deerflow.tools.mcp_metadata import is_mcp_tool
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Checksum validators — the deterministic gate for numeric identifiers.
# ---------------------------------------------------------------------------
def _luhn_valid(value: str) -> bool:
"""Luhn checksum over the digits of *value* (separators ignored)."""
digits = [int(ch) for ch in value if ch.isdigit()]
if len(digits) < 13 or len(digits) > 19:
return False
checksum = 0
for offset, digit in enumerate(reversed(digits)):
if offset % 2 == 1:
digit *= 2
if digit > 9:
digit -= 9
checksum += digit
return checksum % 10 == 0
_CN_ID_WEIGHTS = (7, 9, 10, 5, 8, 4, 2, 1, 6, 3, 7, 9, 10, 5, 8, 4, 2)
_CN_ID_CHECK_DIGITS = "10X98765432"
def _cn_resident_id_valid(value: str) -> bool:
"""GB 11643 checksum for the 18-digit resident ID number."""
body = value[:17]
if not body.isdigit():
return False
year, month, day = int(body[6:10]), int(body[10:12]), int(body[12:14])
if not (1900 <= year <= 2100 and 1 <= month <= 12 and 1 <= day <= 31):
return False
total = sum(int(digit) * weight for digit, weight in zip(body, _CN_ID_WEIGHTS))
return _CN_ID_CHECK_DIGITS[total % 11] == value[17].upper()
def _cpf_valid(value: str) -> bool:
"""Brazilian CPF mod-11 verification digits."""
digits = [int(ch) for ch in value if ch.isdigit()]
if len(digits) != 11 or len(set(digits)) == 1:
return False
for boundary in (9, 10):
weight = 2
total = 0
for digit in reversed(digits[:boundary]):
total += digit * weight
weight += 1
rest = (total * 10) % 11 % 10
if rest != digits[boundary]:
return False
return True
def _national_id_valid(value: str) -> bool:
"""Dispatch by shape: CN resident ID, CPF; CUIT/RFC are format-only."""
if "." in value or (len(value) == 18 and value[:17].isdigit()):
return _cpf_valid(value) if "." in value else _cn_resident_id_valid(value)
return True
# ---------------------------------------------------------------------------
# Detectors. Order is load-bearing: see module docstring.
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class _Detector:
name: str
pattern: re.Pattern[str]
validator: Callable[[str], bool] | None = None
_EMAIL_PATTERN = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(?:\.[A-Za-z0-9-]+)*\.[A-Za-z]{2,}")
_API_KEY_PATTERN = re.compile(
r"\b(?:"
r"sk-[A-Za-z0-9_-]{20,}" # OpenAI-style
r"|AKIA[0-9A-Z]{16}" # AWS access key id
r"|gh[pousr]_[A-Za-z0-9]{30,}" # GitHub token
r"|github_pat_[A-Za-z0-9_]{20,}" # GitHub fine-grained token
r"|xox[baprs]-[A-Za-z0-9-]{10,}" # Slack token
r"|AIza[0-9A-Za-z_-]{35}" # Google API key
r")\b"
)
# Digit-anchored patterns use digit-aware lookarounds instead of Unicode \b:
# CJK characters are word characters, so \b fails between a Chinese label and
# the identifier and the match is lost entirely ("身份证110105…").
_CREDIT_CARD_PATTERN = re.compile(r"(?<!\d)(?:\d{4}[ -]){3}\d{1,7}(?!\d)|(?<!\d)\d{13,19}(?!\d)")
_PHONE_PATTERN = re.compile(
r"(?<!\d)\+\d{1,3}(?:[ \-]?\d{1,4}){3,6}(?!\d)" # international +CC form; single-char separators so a candidate cannot run across a newline
r"|(?<!\d)1[3-9]\d{9}(?!\d)" # CN mobile
r"|\(\d{3}\) ?\d{3}[-.]?\d{4}(?!\d)" # US formatted
)
_NATIONAL_ID_PATTERN = re.compile(
r"(?<![\dXx])\d{17}[\dXx](?![\dXx])" # CN resident ID (checksum-validated)
r"|(?<!\d)\d{3}\.\d{3}\.\d{3}-\d{2}(?!\d)" # CPF (checksum-validated)
r"|(?<!\d)\d{2}-\d{8}-\d(?!\d)" # CUIT, 2+8+1 digits (format-only)
r"|(?<![A-Z0-9Ñ&])[A-ZÑ&]{4}\d{6}[0-9A-Z]{3}(?![A-Z0-9Ñ&])" # RFC with homoclave (format-only)
)
_DETECTORS: tuple[_Detector, ...] = (
_Detector("email", _EMAIL_PATTERN),
_Detector("api_key", _API_KEY_PATTERN),
_Detector("national_id", _NATIONAL_ID_PATTERN, _national_id_valid),
_Detector("credit_card", _CREDIT_CARD_PATTERN, _luhn_valid),
_Detector("phone", _PHONE_PATTERN, lambda value: 8 <= len(re.sub(r"\D", "", value)) <= 15),
)
def active_pii_detectors(config: PiiRedactionConfig | None) -> tuple[_Detector, ...]:
"""Detectors active under *config*; empty when the feature is off."""
if config is None or not config.enabled:
return ()
return tuple(d for d in _DETECTORS if getattr(config, f"redact_{d.name}"))
def redact_text(text: str | None, config: PiiRedactionConfig | None) -> str | None:
"""Redact PII from *text* under *config*; ``None``/disabled leaves it unchanged.
Shared entry point for the seams that sit *outside* this middleware's
``wrap_model_call`` wrapper — SummarizationMiddleware invokes its summary
model directly from ``before_model``, and DurableContextMiddleware injects
its durable-context block inner of it — so compaction input and reinjected
summaries get the same treatment as model-bound messages.
"""
if config is None or not config.enabled or not isinstance(text, str) or not text:
return text
return _Redactor(active_pii_detectors(config)).redact(text)
def redact_texts(texts: Sequence[str], config: PiiRedactionConfig | None) -> list[str]:
"""Redact related text fields with one allocation scope, before truncation."""
redactor = _Redactor(active_pii_detectors(config))
for text in texts:
redactor.reserve(text)
return [redactor.redact(text) for text in texts]
# Bound the numeric field before int() conversion; generated indices are tiny.
_PLACEHOLDER_PATTERN = re.compile(r"\[(EMAIL|API_KEY|NATIONAL_ID|CREDIT_CARD|PHONE)_([1-9][0-9]{0,19})\]")
class _Redactor:
"""Per-scan redaction state: one stable placeholder per distinct value.
A single instance covers one scan (one model request, or one tool result),
so identical raw values in that scan render the same placeholder. Existing
tokens reserve indices but cannot recover compacted raw-value identities. Placeholders are irreversible — the mapping lives only as
long as this instance.
"""
def __init__(self, detectors: Sequence[_Detector]) -> None:
self._detectors = detectors
self._tokens: dict[tuple[str, str], str] = {}
self._counts: dict[str, int] = {}
def reserve(self, content: object) -> None:
"""Reserve visible placeholders before assigning any new values.
Only placeholder counters survive compaction, never raw-value mappings.
Pre-scan all fields so a token in a later block cannot collide with a
new value in an earlier one.
"""
if isinstance(content, str):
for match in _PLACEHOLDER_PATTERN.finditer(content):
name, index = match.groups()
self._counts[name.lower()] = max(self._counts.get(name.lower(), 0), int(index))
elif isinstance(content, list):
for block in content:
if isinstance(block, str):
self.reserve(block)
elif isinstance(block, dict) and block.get("type") == "text":
self.reserve(block.get("text"))
def redact(self, text: str) -> str:
self.reserve(text)
for detector in self._detectors:
text = detector.pattern.sub(self._replacer(detector), text)
return text
def _replacer(self, detector: _Detector) -> Callable[[re.Match[str]], str]:
def replace(match: re.Match[str]) -> str:
value = match.group(0)
if detector.validator is not None and not detector.validator(value):
return value
key = (detector.name, value)
token = self._tokens.get(key)
if token is None:
self._counts[detector.name] = self._counts.get(detector.name, 0) + 1
token = f"[{detector.name.upper()}_{self._counts[detector.name]}]"
self._tokens[key] = token
return token
return replace
def _redact_content(content: object, redactor: _Redactor) -> tuple[object, bool]:
"""Redact *content*, preserving its shape. Returns ``(content, changed)``.
Handles the two shapes message content takes — plain ``str`` and a list of
content blocks. Non-text blocks (images, etc.) pass through untouched.
The input is never mutated.
"""
redactor.reserve(content)
if isinstance(content, str):
redacted = redactor.redact(content)
return redacted, redacted != content
if not isinstance(content, list):
return content, False
new_content: list = []
changed = False
for block in content:
if isinstance(block, str):
redacted = redactor.redact(block)
changed = changed or redacted != block
new_content.append(redacted)
elif isinstance(block, dict) and block.get("type") == "text" and isinstance(block.get("text"), str):
redacted = redactor.redact(block["text"])
if redacted != block["text"]:
new_content.append({**block, "text": redacted})
changed = True
else:
new_content.append(block)
else:
new_content.append(block)
return new_content, changed
class PiiRedactionMiddleware(AgentMiddleware[AgentState]):
"""Rewrite PII in user messages and remote tool results to placeholders.
Assembled only when ``pii_redaction.enabled`` is true, so every instance
has at least one active detector. Unexpected errors fail open (the original
content reaches the model) — consistent with the other guardrails, where
one unprocessable row must not break the run; the trade-off is logged.
"""
def __init__(self, config: PiiRedactionConfig) -> None:
self._detectors = active_pii_detectors(config)
def release_policy_parameters(self) -> dict[str, object]:
"""Declare the behaviour-affecting settings (middleware module guide)."""
return {
"enabled": True,
"detectors": sorted(detector.name for detector in self._detectors),
}
# -- model-call boundary: genuine user messages ---------------------------
def _process_request(self, request: ModelRequest) -> ModelRequest:
redactor = _Redactor(self._detectors)
messages = list(request.messages)
state = getattr(request, "state", None) or {}
summary = state.get("summary_text")
redactor.reserve(summary)
for message in messages:
redactor.reserve(message.content)
redacted_summary = redactor.redact(summary) if isinstance(summary, str) else summary
changed = False
for index, msg in enumerate(messages):
if not isinstance(msg, HumanMessage) or not requires_input_sanitization(msg):
continue
try:
content, changed_msg = _redact_content(msg.content, redactor)
except GraphBubbleUp:
raise
except Exception:
logger.warning(
"PII redaction failed on user message at pos=%d; leaving it unchanged",
index,
exc_info=True,
)
continue
if not changed_msg:
continue
messages[index] = HumanMessage(
content=content,
id=msg.id,
name=msg.name,
additional_kwargs=dict(msg.additional_kwargs or {}),
)
changed = True
updates = {}
if changed:
updates["messages"] = messages
if redacted_summary != summary:
# Request-local copy only: the inner durable-context wrapper must
# use the same allocation as the retained user messages.
updates["state"] = {**state, "summary_text": redacted_summary}
return request.override(**updates) if updates else request
def _try_process(self, request: ModelRequest) -> ModelRequest:
try:
return self._process_request(request)
except GraphBubbleUp:
raise
except Exception:
logger.warning("PII redaction processing failed; passing original request to model", exc_info=True)
return request
@override
def wrap_model_call(
self,
request: ModelRequest,
handler: Callable[[ModelRequest], ModelResponse],
) -> ModelCallResult:
return handler(self._try_process(request))
@override
async def awrap_model_call(
self,
request: ModelRequest,
handler: Callable[[ModelRequest], Awaitable[ModelResponse]],
) -> ModelCallResult:
return await handler(self._try_process(request))
# -- tool boundary: remote-content tool results ---------------------------
def _should_redact(self, request: ToolCallRequest) -> bool:
if request.tool_call.get("name") in _REMOTE_CONTENT_TOOL_NAMES:
return True
return is_mcp_tool(getattr(request, "tool", None))
def _redact_result(self, result: ToolMessage | Command) -> ToolMessage | Command:
"""Redact a tool-call result, mirroring ``_sanitize_result``'s shapes.
Direct ``ToolMessage`` results are redacted; ``Command`` results carry
their ToolMessages inside ``update.messages`` and are rebuilt with
``dc_replace`` only when one of them actually changed. One redactor
spans the whole result, so placeholder numbering stays continuous
across every ToolMessage the result carries.
"""
redactor = _Redactor(self._detectors)
if isinstance(result, ToolMessage):
return self._redact_tool_message(result, redactor)
update = getattr(result, "update", None)
if isinstance(update, dict):
messages = update.get("messages")
if isinstance(messages, list) and any(isinstance(m, ToolMessage) for m in messages):
for message in messages:
if isinstance(message, ToolMessage):
redactor.reserve(message.content)
new_messages = [self._redact_tool_message(m, redactor) if isinstance(m, ToolMessage) else m for m in messages]
if new_messages != messages:
return dc_replace(result, update={**update, "messages": new_messages})
return result
def _redact_tool_message(self, message: ToolMessage, redactor: _Redactor) -> ToolMessage:
content, changed = _redact_content(message.content, redactor)
if not changed:
return message
additional_kwargs = dict(message.additional_kwargs or {})
append_tool_transform(additional_kwargs, "pii_redaction", by="PiiRedactionMiddleware")
# model_copy preserves artifact / response_metadata that a hand-built
# constructor call would silently drop.
return message.model_copy(update={"content": content, "additional_kwargs": additional_kwargs})
@override
def wrap_tool_call(
self,
request: ToolCallRequest,
handler: Callable[[ToolCallRequest], ToolMessage | Command],
) -> ToolMessage | Command:
result = handler(request)
if not self._should_redact(request):
return result
try:
return self._redact_result(result)
except Exception:
logger.warning("PII redaction failed on tool result; leaving it unchanged", exc_info=True)
return result
@override
async def awrap_tool_call(
self,
request: ToolCallRequest,
handler: Callable[[ToolCallRequest], Awaitable[ToolMessage | Command]],
) -> ToolMessage | Command:
result = await handler(request)
if not self._should_redact(request):
return result
try:
return self._redact_result(result)
except Exception:
logger.warning("PII redaction failed on tool result; leaving it unchanged", exc_info=True)
return result

6
backend/packages/harness/deerflow/agents/middlewares/summarization_middleware.py
View File

@ -19,6 +19,7 @@ from langgraph.runtime import Runtime
from deerflow.agents.middlewares.dynamic_context_middleware import is_dynamic_context_reminder
from deerflow.agents.middlewares.message_utils import is_genuine_user_message
from deerflow.agents.middlewares.pii_redaction_middleware import redact_text
from deerflow.config.app_config import get_app_config
from deerflow.config.summarization_config import DEFAULT_KEEP
from deerflow.config.task_continuity_config import TaskContinuityConfig
@ -547,6 +548,11 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
formatted_messages = self._build_summary_input_text(formatted_messages, previous_summary=previous_summary, new_messages_strategy=new_messages_strategy)
if not formatted_messages:
return None
# The summary model is invoked directly from before_model, outside
# PiiRedactionMiddleware's wrap_model_call (#3190), so the compaction
# input is redacted here; summaries then carry placeholders and the
# summary_text DurableContextMiddleware reinjects stays clean.
formatted_messages = redact_text(formatted_messages, getattr(self._app_config, "pii_redaction", None))
return self.summary_prompt.format(messages=formatted_messages).rstrip()
def before_model(self, state: AgentState, runtime: Runtime) -> dict | None:

8
backend/packages/harness/deerflow/agents/middlewares/title_middleware.py
View File

@ -14,6 +14,7 @@ from langgraph.constants import TAG_NOSTREAM
from langgraph.runtime import Runtime
from deerflow.agents.middlewares.dynamic_context_middleware import is_dynamic_context_reminder
from deerflow.agents.middlewares.pii_redaction_middleware import redact_texts
from deerflow.config.title_config import get_title_config
from deerflow.models import create_chat_model
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY, get_original_user_content_text
@ -215,10 +216,13 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
user_msg = self._get_title_user_message(state)
assistant_msg = self._strip_think_tags(self._normalize_content(assistant_msg_content))
# This model is invoked directly, outside the main model wrappers.
# Redact complete fields before truncation can split an identifier.
redacted_user, redacted_assistant = redact_texts([user_msg, assistant_msg], getattr(self._app_config, "pii_redaction", None))
prompt = config.prompt_template.format(
max_words=config.max_words,
user_msg=user_msg[:500],
assistant_msg=assistant_msg[:500],
user_msg=redacted_user[:500],
assistant_msg=redacted_assistant[:500],
)
return prompt, user_msg

10
backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py
View File

@ -194,6 +194,15 @@ def _build_runtime_middlewares(
ToolOutputBudgetMiddleware.from_app_config(app_config),
ToolResultSanitizationMiddleware(),
]
if app_config.pii_redaction.enabled:
from deerflow.agents.middlewares.pii_redaction_middleware import PiiRedactionMiddleware
# Listed last so it is the innermost Layer-1 wrapper: tool results are
# PII-redacted before ToolResultSanitizationMiddleware neutralizes tags
# and ToolOutputBudgetMiddleware externalizes oversized copies to disk
# (so those copies hold redacted text), and user messages reach it
# after the other request rewrites (issue #3190).
outer_wrappers.append(PiiRedactionMiddleware(app_config.pii_redaction))
# Layer 2 — before_agent hooks that read/annotate thread-scoped data.
thread_hooks: list[AgentMiddleware] = [
@ -509,6 +518,7 @@ def build_subagent_runtime_middlewares(
DurableContextMiddleware(
skills_container_path=app_config.skills.container_path,
skill_file_read_tool_names=app_config.summarization.skill_file_read_tool_names,
pii_redaction_config=getattr(app_config, "pii_redaction", None),
)
)

2
backend/packages/harness/deerflow/config/app_config.py
View File

@ -28,6 +28,7 @@ from deerflow.config.loop_detection_config import LoopDetectionConfig
from deerflow.config.mcp_tasks_config import McpTasksConfig
from deerflow.config.memory_config import MemoryConfig, load_memory_config_from_dict
from deerflow.config.model_config import ModelConfig
from deerflow.config.pii_redaction_config import PiiRedactionConfig
from deerflow.config.projects_config import ProjectsConfig
from deerflow.config.read_before_write_config import ReadBeforeWriteConfig
from deerflow.config.reload_boundary import format_field_description
@ -271,6 +272,7 @@ class AppConfig(BaseModel):
verification: VerificationConfig = Field(default_factory=VerificationConfig, description="Subagent result verification (receipts, checklist, judge)")
read_before_write: ReadBeforeWriteConfig = Field(default_factory=ReadBeforeWriteConfig, description="Read-before-write file gate middleware configuration")
projects: ProjectsConfig = Field(default_factory=ProjectsConfig, description="User projects configuration (instructions injection, shelf index, trash retention)")
pii_redaction: PiiRedactionConfig = Field(default_factory=PiiRedactionConfig, description="PII redaction middleware configuration (issue #3190)")
safety_finish_reason: SafetyFinishReasonConfig = Field(default_factory=SafetyFinishReasonConfig, description="Provider safety-filter finish_reason interception middleware configuration")
auth: AuthAppConfig = Field(default_factory=AuthAppConfig, description="Authentication configuration (local + OIDC SSO)")
model_config = ConfigDict(extra="allow")

39
backend/packages/harness/deerflow/config/pii_redaction_config.py Normal file
View File

@ -0,0 +1,39 @@
"""Configuration for the PII redaction middleware (issue #3190)."""
from pydantic import BaseModel, Field
class PiiRedactionConfig(BaseModel):
"""Configuration for deterministic PII redaction in model-bound context.
Default-off. When enabled, personally identifiable information found in
genuine user messages and remote-content tool results is rewritten to
irreversible placeholders (``[EMAIL_1]`` …) before it reaches the model.
Each detector can be toggled independently for deployments that only need
a subset (e.g. credentials but not phone numbers).
"""
enabled: bool = Field(
default=False,
description="Whether to enable PII redaction in model-bound context",
)
redact_email: bool = Field(
default=True,
description="Redact email addresses",
)
redact_api_key: bool = Field(
default=True,
description="Redact API keys and bearer-style tokens (OpenAI sk-, AWS AKIA, GitHub ghp_/github_pat_, Slack xox-, Google AIza)",
)
redact_credit_card: bool = Field(
default=True,
description="Redact credit-card numbers passing the Luhn checksum",
)
redact_phone: bool = Field(
default=True,
description="Redact phone numbers (international +CC form, CN mobile, US formatted)",
)
redact_national_id: bool = Field(
default=True,
description="Redact national IDs (CN resident ID and CPF with checksum validation, formatted CUIT/RFC)",
)

666
backend/tests/test_pii_redaction_middleware.py Normal file
View File

@ -0,0 +1,666 @@
"""Tests for PiiRedactionMiddleware (issue #3190).
Verifies deterministic detector coverage (including checksum gates), stable
placeholder numbering across a conversation, that the rewrite is request-scoped
without mutating the original request or messages, the tool-boundary allowlist,
and the pinned detector registry.
"""
import re
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, Mock
import pytest
from _agent_e2e_helpers import FakeToolCallingModel
from langchain.agents import create_agent
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage, get_buffer_string
from langchain_core.outputs import ChatGeneration, ChatResult
from langgraph.types import Command
from pydantic import Field
from deerflow.agents.middlewares.pii_redaction_middleware import (
_DETECTORS,
PiiRedactionMiddleware,
redact_text,
)
from deerflow.config.pii_redaction_config import PiiRedactionConfig
from deerflow.tools.mcp_metadata import MCP_TOOL_METADATA_KEY
def _make_middleware(**config_overrides) -> PiiRedactionMiddleware:
return PiiRedactionMiddleware(PiiRedactionConfig(enabled=True, **config_overrides))
class _FakeRequest:
"""Minimal stand-in for ModelRequest — duck-typed to .messages + .override()."""
def __init__(self, messages):
self.messages = list(messages)
def override(self, **kwargs):
return _FakeRequest(kwargs.get("messages", self.messages))
def _run_model_call(middleware, messages):
"""Run wrap_model_call; return (final_messages, original_request)."""
request = _FakeRequest(messages)
captured = {}
middleware.wrap_model_call(request, lambda req: captured.update(messages=req.messages) or "response")
return captured["messages"], request
def _run_tool_call(middleware, tool_name, result, *, tool=None):
request = Mock()
request.tool_call = {"name": tool_name}
request.tool = tool if tool is not None else SimpleNamespace(metadata=None)
return middleware.wrap_tool_call(request, lambda _request: result)
# ---------------------------------------------------------------------------
# Detectors
# ---------------------------------------------------------------------------
class TestDetectors:
def test_pinned_detector_count(self):
"""New detectors must extend this pin and the config toggles together."""
assert len(_DETECTORS) == 5
assert [d.name for d in _DETECTORS] == ["email", "api_key", "national_id", "credit_card", "phone"]
def test_cn_resident_id_with_luhn_valid_digits_not_mislabeled_as_card(self):
# 110105194912310150 passes both the GB 11643 checksum and Luhn; the
# national-id detector must claim it before the credit-card detector
# (review finding on #5527, reproduced at 0a2a9d0).
messages, _ = _run_model_call(_make_middleware(), [HumanMessage("id 110105194912310150")])
assert "id [NATIONAL_ID_1]" in messages[0].content
def test_email_redacted(self):
result = _make_middleware()._detectors[0].pattern.sub("X", "ping me at alice@example.com today")
assert result == "ping me at X today"
def test_distinct_emails_get_distinct_placeholders(self):
middleware = _make_middleware()
messages, _ = _run_model_call(
middleware,
[HumanMessage("from alice@example.com to bob@example.org")],
)
assert "from [EMAIL_1] to [EMAIL_2]" in messages[0].content
def test_same_email_shares_placeholder(self):
middleware = _make_middleware()
messages, _ = _run_model_call(
middleware,
[
HumanMessage("alice@example.com here"),
HumanMessage("reply to alice@example.com"),
],
)
assert messages[0].content == "[EMAIL_1] here"
assert messages[1].content == "reply to [EMAIL_1]"
def test_openai_style_api_key_redacted(self):
messages, _ = _run_model_call(
_make_middleware(),
[HumanMessage("key: sk-proj4aaaaaaaaaaaaaaaaaaaaaaaaaaaa")],
)
assert "[API_KEY_1]" in messages[0].content
def test_aws_access_key_redacted(self):
messages, _ = _run_model_call(
_make_middleware(),
[HumanMessage("use AKIAIOSFODNN7EXAMPLE please")],
)
assert "[API_KEY_1]" in messages[0].content
def test_credit_card_luhn_valid_redacted(self):
messages, _ = _run_model_call(
_make_middleware(),
[HumanMessage("card 4111 1111 1111 1111 on file")],
)
assert "card [CREDIT_CARD_1] on file" in messages[0].content
def test_credit_card_luhn_invalid_untouched(self):
original = "card 1234 5678 9012 3456 on file"
messages, _ = _run_model_call(_make_middleware(), [HumanMessage(original)])
assert messages[0].content == original
def test_long_digit_run_non_card_untouched(self):
original = "order 1234567890123 shipped"
messages, _ = _run_model_call(_make_middleware(), [HumanMessage(original)])
assert messages[0].content == original
def test_international_phone_redacted(self):
messages, _ = _run_model_call(
_make_middleware(),
[HumanMessage("call +86 138 0013 8000 now")],
)
assert "call [PHONE_1] now" in messages[0].content
def test_cn_mobile_redacted(self):
messages, _ = _run_model_call(_make_middleware(), [HumanMessage("phone 13800138000")])
assert "phone [PHONE_1]" in messages[0].content
def test_us_phone_redacted(self):
messages, _ = _run_model_call(_make_middleware(), [HumanMessage("dial (212) 555-0123")])
assert "dial [PHONE_1]" in messages[0].content
def test_cn_resident_id_valid_redacted(self):
messages, _ = _run_model_call(
_make_middleware(),
[HumanMessage("id 11010519491231002X")],
)
assert "id [NATIONAL_ID_1]" in messages[0].content
def test_cn_resident_id_invalid_checksum_untouched(self):
original = "id 110105194912310020"
messages, _ = _run_model_call(_make_middleware(), [HumanMessage(original)])
assert messages[0].content == original
def test_willem_vector_numeric_check_digit_redacted(self):
# Review vector on #5527: numeric-check-digit resident ID whose digits
# also pass Luhn must render NATIONAL_ID, not CREDIT_CARD.
messages, _ = _run_model_call(_make_middleware(), [HumanMessage("id 110105197506150239")])
assert "id [NATIONAL_ID_1]" in messages[0].content
def test_cuit_valid_form_redacted(self):
messages, _ = _run_model_call(_make_middleware(), [HumanMessage("CUIT 20-12345678-6")])
assert "CUIT [NATIONAL_ID_1]" in messages[0].content
def test_cuit_wrong_digit_count_untouched(self):
original = "CUIT 20-1234567890-6"
messages, _ = _run_model_call(_make_middleware(), [HumanMessage(original)])
assert messages[0].content == original
def test_cjk_adjacent_identifiers_redacted(self):
# Python \b treats CJK as word characters; the digit-aware lookarounds
# must still catch identifiers glued to Chinese labels.
messages, _ = _run_model_call(
_make_middleware(),
[HumanMessage("身份证11010519491231002X 手机号13800138000 信用卡4111 1111 1111 1111")],
)
content = messages[0].content
assert "[NATIONAL_ID_1]" in content and "[PHONE_1]" in content and "[CREDIT_CARD_1]" in content
assert "11010519491231002X" not in content and "13800138000" not in content and "4111" not in content
def test_international_phone_does_not_consume_next_line(self):
messages, _ = _run_model_call(
_make_middleware(),
[HumanMessage("Call +1 415 555 2671\n20260918")],
)
assert messages[0].content == "Call [PHONE_1]\n20260918"
def test_cpf_valid_redacted(self):
messages, _ = _run_model_call(
_make_middleware(),
[HumanMessage("cpf 529.982.247-25")],
)
assert "cpf [NATIONAL_ID_1]" in messages[0].content
def test_cpf_invalid_untouched(self):
original = "cpf 529.982.247-11"
messages, _ = _run_model_call(_make_middleware(), [HumanMessage(original)])
assert messages[0].content == original
# ---------------------------------------------------------------------------
# Model-call boundary
# ---------------------------------------------------------------------------
class TestModelCallBoundary:
def test_genuine_user_message_redacted(self):
messages, _ = _run_model_call(
_make_middleware(),
[HumanMessage("my email is alice@example.com")],
)
assert messages[0].content == "my email is [EMAIL_1]"
def test_original_request_not_mutated(self):
original = HumanMessage("my email is alice@example.com")
messages, request = _run_model_call(_make_middleware(), [original])
assert messages[0].content == "my email is [EMAIL_1]"
assert request.messages[0].content == "my email is alice@example.com"
def test_additional_kwargs_preserved(self):
original = HumanMessage("alice@example.com", additional_kwargs={"hide_from_ui": False, "custom": "v"})
messages, _ = _run_model_call(_make_middleware(), [original])
assert messages[0].additional_kwargs["custom"] == "v"
def test_ai_message_untouched(self):
ai = AIMessage("contact alice@example.com")
messages, _ = _run_model_call(_make_middleware(), [ai])
assert messages[0].content == "contact alice@example.com"
def test_clean_message_not_rebuilt(self):
original = HumanMessage("no secrets here")
messages, _ = _run_model_call(_make_middleware(), [original])
assert messages[0] is original
def test_placeholder_numbering_spans_conversation(self):
messages, _ = _run_model_call(
_make_middleware(),
[
HumanMessage("first alice@example.com"),
AIMessage("noted"),
HumanMessage("then bob@example.org"),
],
)
assert "first [EMAIL_1]" in messages[0].content
assert "then [EMAIL_2]" in messages[2].content
def test_redaction_deterministic_across_calls(self):
middleware = _make_middleware()
messages_a, _ = _run_model_call(middleware, [HumanMessage("alice@example.com")])
messages_b, _ = _run_model_call(middleware, [HumanMessage("alice@example.com")])
assert messages_a[0].content == messages_b[0].content == "[EMAIL_1]"
def test_disabled_detector_untouched(self):
messages, _ = _run_model_call(
_make_middleware(redact_email=False),
[HumanMessage("alice@example.com")],
)
assert messages[0].content == "alice@example.com"
def test_multimodal_text_blocks_redacted_and_non_text_kept(self):
image_block = {"type": "image_url", "image_url": {"url": "https://example.com/a.png"}}
original = HumanMessage(
[
"reach me at alice@example.com",
image_block,
"or bob@example.org",
]
)
messages, _ = _run_model_call(_make_middleware(), [original])
assert messages[0].content[0] == "reach me at [EMAIL_1]"
# LangChain rebuilds content blocks on construction, so compare by value.
assert messages[0].content[1] == image_block
assert messages[0].content[2] == "or [EMAIL_2]"
# The original message object is untouched.
assert original.content[0] == "reach me at alice@example.com"
# ---------------------------------------------------------------------------
# Tool boundary
# ---------------------------------------------------------------------------
class TestToolBoundary:
def test_web_fetch_result_redacted_and_stamped(self):
result = ToolMessage(
content="page says contact alice@example.com",
tool_call_id="call_1",
name="web_fetch",
)
final = _run_tool_call(_make_middleware(), "web_fetch", result)
assert final.content == "page says contact [EMAIL_1]"
transforms = final.additional_kwargs["deerflow_tool_transforms"]
assert transforms[-1]["kind"] == "pii_redaction"
assert transforms[-1]["by"] == "PiiRedactionMiddleware"
def test_local_tool_result_untouched(self):
result = ToolMessage(
content="user row: alice@example.com",
tool_call_id="call_1",
name="bash",
)
final = _run_tool_call(_make_middleware(), "bash", result)
assert final is result
def test_mcp_tagged_tool_redacted(self):
result = ToolMessage(
content="alice@example.com",
tool_call_id="call_1",
name="fetch_url",
)
tool = SimpleNamespace(metadata={MCP_TOOL_METADATA_KEY: True})
final = _run_tool_call(_make_middleware(), "fetch_url", result, tool=tool)
assert final.content == "[EMAIL_1]"
def test_command_result_passthrough(self):
result = Command(update={"events": ["alice@example.com"]})
final = _run_tool_call(_make_middleware(), "web_fetch", result)
assert final is result
def test_command_wrapped_tool_result_redacted_and_stamped(self):
tool_message = ToolMessage(content="page says alice@example.com", tool_call_id="c1", name="web_fetch")
result = Command(update={"messages": [tool_message]})
final = _run_tool_call(_make_middleware(), "web_fetch", result)
assert isinstance(final, Command)
new_message = final.update["messages"][0]
assert new_message.content == "page says [EMAIL_1]"
assert new_message.additional_kwargs["deerflow_tool_transforms"][-1]["kind"] == "pii_redaction"
# The original Command and its message are untouched.
assert tool_message.content == "page says alice@example.com"
def test_command_without_tool_messages_passthrough(self):
result = Command(update={"messages": [AIMessage("alice@example.com")]})
final = _run_tool_call(_make_middleware(), "web_fetch", result)
assert final is result
def test_redacted_tool_message_preserves_artifact_and_metadata(self):
result = ToolMessage(
content="alice@example.com",
tool_call_id="c1",
name="web_fetch",
artifact={"rows": 3},
response_metadata={"latency_ms": 12},
)
final = _run_tool_call(_make_middleware(), "web_fetch", result)
assert final.content == "[EMAIL_1]"
assert final.artifact == {"rows": 3}
assert final.response_metadata == {"latency_ms": 12}
assert final.status == "success"
def test_tool_message_not_mutated(self):
result = ToolMessage(
content="alice@example.com",
tool_call_id="call_1",
name="web_search",
)
_run_tool_call(_make_middleware(), "web_search", result)
assert result.content == "alice@example.com"
def test_command_placeholder_numbering_continues_across_messages(self):
# One redactor spans the whole Command result, so placeholder numbers
# stay continuous across the ToolMessages it carries (review follow-up).
first = ToolMessage(content="alice@example.com", tool_call_id="c1", name="web_fetch")
second = ToolMessage(content="then bob@example.org and alice@example.com", tool_call_id="c2", name="web_fetch")
result = Command(update={"messages": [first, second]})
final = _run_tool_call(_make_middleware(), "web_fetch", result)
messages = final.update["messages"]
assert messages[0].content == "[EMAIL_1]"
assert messages[1].content == "then [EMAIL_2] and [EMAIL_1]"
def test_placeholder_restarts_per_result(self):
middleware = _make_middleware()
first = _run_tool_call(
middleware,
"web_fetch",
ToolMessage(content="alice@example.com", tool_call_id="c1", name="web_fetch"),
)
second = _run_tool_call(
middleware,
"web_fetch",
ToolMessage(content="bob@example.com", tool_call_id="c2", name="web_fetch"),
)
assert first.content == "[EMAIL_1]"
assert second.content == "[EMAIL_1]"
# ---------------------------------------------------------------------------
# release policy declaration
# ---------------------------------------------------------------------------
class TestReleasePolicy:
def test_declares_enabled_detectors(self):
policy = _make_middleware(redact_phone=False, redact_national_id=False).release_policy_parameters()
assert policy == {"enabled": True, "detectors": ["api_key", "credit_card", "email"]}
def test_all_detectors_enabled_by_default_config(self):
policy = _make_middleware().release_policy_parameters()
assert policy["detectors"] == ["api_key", "credit_card", "email", "national_id", "phone"]
@pytest.mark.parametrize(
"config",
[
PiiRedactionConfig(enabled=False),
PiiRedactionConfig(enabled=True),
],
)
def test_config_defaults_are_consistent(config):
"""The middleware constructor must accept the shipped default configs."""
PiiRedactionMiddleware(config)
# ---------------------------------------------------------------------------
# Chain wiring
# ---------------------------------------------------------------------------
def _wiring_app_config(**overrides):
from deerflow.config.app_config import AppConfig
from deerflow.config.sandbox_config import SandboxConfig
return AppConfig(sandbox=SandboxConfig(use="test"), **overrides)
class TestChainWiring:
def test_disabled_by_default_not_in_chain(self):
from deerflow.agents.middlewares.pii_redaction_middleware import PiiRedactionMiddleware
from deerflow.agents.middlewares.tool_error_handling_middleware import build_lead_runtime_middlewares
middlewares = build_lead_runtime_middlewares(app_config=_wiring_app_config())
assert PiiRedactionMiddleware not in [type(m) for m in middlewares]
def test_enabled_sits_inner_of_the_structural_guardrails(self):
from deerflow.agents.middlewares.input_sanitization_middleware import InputSanitizationMiddleware
from deerflow.agents.middlewares.pii_redaction_middleware import PiiRedactionMiddleware
from deerflow.agents.middlewares.tool_error_handling_middleware import build_lead_runtime_middlewares
from deerflow.agents.middlewares.tool_result_sanitization_middleware import ToolResultSanitizationMiddleware
middlewares = build_lead_runtime_middlewares(
app_config=_wiring_app_config(pii_redaction=PiiRedactionConfig(enabled=True)),
)
types = [type(m) for m in middlewares]
assert PiiRedactionMiddleware in types
assert types.index(InputSanitizationMiddleware) < types.index(ToolResultSanitizationMiddleware) < types.index(PiiRedactionMiddleware)
def test_enabled_reaches_subagent_chain(self):
from deerflow.agents.middlewares.pii_redaction_middleware import PiiRedactionMiddleware
from deerflow.agents.middlewares.tool_error_handling_middleware import build_subagent_runtime_middlewares
middlewares = build_subagent_runtime_middlewares(
app_config=_wiring_app_config(pii_redaction=PiiRedactionConfig(enabled=True)),
)
assert PiiRedactionMiddleware in [type(m) for m in middlewares]
# ---------------------------------------------------------------------------
# Shared seams: compaction input + durable-context reinjection (#3190 review)
# ---------------------------------------------------------------------------
class TestRedactTextSharedSeam:
def test_none_config_returns_text_unchanged(self):
assert redact_text("alice@example.com", None) == "alice@example.com"
def test_disabled_config_returns_text_unchanged(self):
assert redact_text("alice@example.com", PiiRedactionConfig(enabled=False)) == "alice@example.com"
def test_enabled_config_redacts(self):
assert redact_text("call alice@example.com", PiiRedactionConfig(enabled=True)) == "call [EMAIL_1]"
def test_non_string_passthrough(self):
assert redact_text(None, PiiRedactionConfig(enabled=True)) is None
class _StateRequest:
"""Duck-typed ModelRequest carrying .state, .messages and .override()."""
def __init__(self, state, messages):
self.state = state
self.messages = list(messages)
def override(self, **kwargs):
copy = object.__new__(type(self))
copy.state = kwargs.get("state", self.state)
copy.messages = kwargs.get("messages", self.messages)
return copy
class TestDurableContextReinjection:
def _make_dc(self, config):
from deerflow.agents.middlewares.durable_context_middleware import DurableContextMiddleware
return DurableContextMiddleware(pii_redaction_config=config)
def test_reinjected_summary_redacted(self):
mw = self._make_dc(PiiRedactionConfig(enabled=True))
request = _StateRequest({"summary_text": "summary of alice@example.com"}, [HumanMessage("hi")])
final = mw._inject(request)
# insert_after_leading_system_messages puts the injected pair up front:
# [authority SystemMessage, durable-context data block, original…].
block = final.messages[1].content
assert "[EMAIL_1]" in block and "alice@example.com" not in block
def test_reinjected_summary_untouched_without_config(self):
mw = self._make_dc(None)
request = _StateRequest({"summary_text": "summary of alice@example.com"}, [HumanMessage("hi")])
final = mw._inject(request)
assert "alice@example.com" in final.messages[1].content
def test_policy_declares_pii_gate(self):
enabled = self._make_dc(PiiRedactionConfig(enabled=True)).release_policy_parameters()
disabled = self._make_dc(None).release_policy_parameters()
assert enabled["pii_redaction_enabled"] is True
assert disabled["pii_redaction_enabled"] is False
class TestSummarizationCompactionInput:
def _middleware(self, pii_config):
from deerflow.agents.middlewares.summarization_middleware import DeerFlowSummarizationMiddleware
model = MagicMock()
model.invoke.return_value = SimpleNamespace(text="compressed")
model.ainvoke = AsyncMock(return_value=SimpleNamespace(text="compressed"))
model.with_config.return_value = model
return DeerFlowSummarizationMiddleware(
model=model,
trigger=("messages", 4),
keep=("messages", 2),
token_counter=len,
app_config=SimpleNamespace(pii_redaction=pii_config),
)
def test_compaction_input_redacted(self):
mw = self._middleware(PiiRedactionConfig(enabled=True))
prompt = mw._build_summary_prompt([HumanMessage("reach alice@example.com")], previous_summary=None)
assert prompt is not None
assert "[EMAIL_1]" in prompt and "alice@example.com" not in prompt
def test_compaction_input_untouched_when_disabled(self):
mw = self._middleware(PiiRedactionConfig(enabled=False))
prompt = mw._build_summary_prompt([HumanMessage("reach alice@example.com")], previous_summary=None)
assert "alice@example.com" in prompt
class _RecordingPiiModel(FakeToolCallingModel):
seen: list[str] = Field(default_factory=list)
echo_summary: bool = False
def _generate(self, messages, stop=None, run_manager=None, **kwargs):
text = get_buffer_string(messages)
self.seen.append(text)
if self.echo_summary:
# Preserve the exact placeholder received, rather than inventing one.
token = re.search(r"\[EMAIL_[0-9]+\]", text).group(0)
return ChatResult(generations=[ChatGeneration(message=AIMessage(content=f"Alice's email is {token}"))])
return super()._generate(messages, stop=stop, run_manager=run_manager, **kwargs)
@pytest.mark.asyncio
@pytest.mark.parametrize("enabled", [True, False])
async def test_async_graph_redacts_configured_title_model_input(monkeypatch, enabled):
from deerflow.agents.middlewares.title_middleware import TitleMiddleware
from deerflow.agents.thread_state import ThreadState
from deerflow.config.title_config import TitleConfig
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY
pii = PiiRedactionConfig(enabled=enabled)
config = _wiring_app_config(pii_redaction=pii, title=TitleConfig(enabled=True, model_name="title-model"))
primary = _RecordingPiiModel(responses=[AIMessage(content="Reply to charlie@example.net")])
title = Mock(ainvoke=AsyncMock(return_value=AIMessage(content="Contact records")))
monkeypatch.setattr("deerflow.agents.middlewares.title_middleware.create_chat_model", lambda **kwargs: title)
graph = create_agent(primary, tools=[], state_schema=ThreadState, middleware=[PiiRedactionMiddleware(pii), TitleMiddleware(app_config=config)])
user = HumanMessage(content="Contact alice@example.com", additional_kwargs={ORIGINAL_USER_CONTENT_KEY: "Contact alice@example.com"})
result = await graph.ainvoke({"messages": [user]})
prompt = title.ainvoke.await_args.args[0]
assert result["title"] == "Contact records"
assert result["messages"][0].content == user.content
if enabled:
assert "alice@example.com" not in primary.seen[0]
assert "alice@example.com" not in prompt
assert "charlie@example.net" not in prompt
assert "[EMAIL_1]" in prompt and "[EMAIL_2]" in prompt
else:
assert "alice@example.com" in primary.seen[0]
assert "alice@example.com" in prompt and "charlie@example.net" in prompt
@pytest.mark.parametrize("async_mode", [False, True])
def test_compiled_graph_keeps_summary_and_retained_pii_distinct(async_mode):
import asyncio
from deerflow.agents.middlewares.durable_context_middleware import DurableContextMiddleware
from deerflow.agents.middlewares.summarization_middleware import DeerFlowSummarizationMiddleware
from deerflow.agents.thread_state import ThreadState
pii = PiiRedactionConfig(enabled=True)
config = _wiring_app_config(pii_redaction=pii)
summary = _RecordingPiiModel(responses=[AIMessage(content="unused")], echo_summary=True)
primary = _RecordingPiiModel(responses=[AIMessage(content="done")])
graph = create_agent(
primary,
tools=[],
state_schema=ThreadState,
middleware=[
PiiRedactionMiddleware(pii),
DurableContextMiddleware(pii_redaction_config=pii),
DeerFlowSummarizationMiddleware(model=summary, trigger=("messages", 4), keep=("messages", 2), token_counter=len, app_config=config),
],
)
messages = [HumanMessage(content="Alice's email is alice@example.com"), AIMessage(content="Noted"), HumanMessage(content="Bob's email is bob@example.com; keep their records separate"), AIMessage(content="Noted")]
state = {"messages": messages}
result = asyncio.run(graph.ainvoke(state)) if async_mode else graph.invoke(state)
assert summary.seen and "alice@example.com" not in summary.seen[0]
assert result["summary_text"] == "Alice's email is [EMAIL_1]"
assert "Alice's email is [EMAIL_1]" in primary.seen[0]
assert "Bob's email is [EMAIL_2]" in primary.seen[0]
assert "bob@example.com" not in primary.seen[0]
assert any(message.content == messages[2].content for message in result["messages"])
def test_summary_redaction_reserves_existing_placeholders():
config = PiiRedactionConfig(enabled=True)
assert redact_text("Alice [EMAIL_1], Bob bob@example.com", config) == "Alice [EMAIL_1], Bob [EMAIL_2]"
def test_existing_placeholder_in_later_content_block_is_reserved_first():
messages, _ = _run_model_call(_make_middleware(), [HumanMessage(content=["bob@example.com", {"type": "text", "text": "Alice [EMAIL_1]"}])])
assert messages[0].content == ["[EMAIL_2]", {"type": "text", "text": "Alice [EMAIL_1]"}]
def test_raw_legacy_summary_and_retained_messages_share_request_allocation():
from deerflow.agents.middlewares.durable_context_middleware import DurableContextMiddleware
pii = PiiRedactionConfig(enabled=True)
state = {"summary_text": "Alice alice@example.com"}
request = _StateRequest(state, [HumanMessage(content="Alice alice@example.com; Bob bob@example.com")])
redacted = PiiRedactionMiddleware(pii)._process_request(request)
final = DurableContextMiddleware(pii_redaction_config=pii)._inject(redacted)
text = get_buffer_string(final.messages)
assert "Alice [EMAIL_1]" in text and "Bob [EMAIL_2]" in text
assert "alice@example.com" not in text and "bob@example.com" not in text
assert state == {"summary_text": "Alice alice@example.com"}
assert request.messages[0].content == "Alice alice@example.com; Bob bob@example.com"
def test_repeated_compaction_reserves_prior_summary_tokens():
middleware = TestSummarizationCompactionInput()._middleware(PiiRedactionConfig(enabled=True))
prompt = middleware._build_summary_prompt([HumanMessage("Carol carol@example.com")], previous_summary="Alice [EMAIL_1], Bob [EMAIL_2]")
assert "Alice [EMAIL_1], Bob [EMAIL_2]" in prompt
assert "Carol [EMAIL_3]" in prompt
def test_title_redacts_identifiers_before_field_truncation():
from deerflow.agents.middlewares.title_middleware import TitleMiddleware
config = _wiring_app_config(pii_redaction=PiiRedactionConfig(enabled=True))
prompt, fallback = TitleMiddleware(app_config=config)._build_title_prompt({"messages": [HumanMessage(content="x " * 246 + "alice@example.com"), AIMessage(content="done")]})
assert "alice" not in prompt
assert fallback.endswith("alice@example.com") # Local display fallback preserves the original user text.

20
config.example.yaml
View File

@ -1320,6 +1320,26 @@ loop_detection:
# warn: 150
# hard_limit: 300
# ============================================================================
# PII Redaction Middleware Configuration (issue #3190)
# ============================================================================
# Deterministic PII redaction in model-bound context. When enabled, emails,
# phone numbers, API keys, credit-card numbers (Luhn-validated) and national
# IDs found in genuine user messages and remote-content tool results
# (web_fetch / web_search / image_search / web_capture and MCP tools) are
# rewritten to irreversible placeholders like [EMAIL_1] before they reach the
# model. No mapping is stored, so redaction cannot be reversed. Raw text stays
# in thread state for the UI; the memory-extraction path is not covered.
# Compaction and configured LLM title inputs are redacted too. Existing
# placeholders reserve indices; compacted raw identities cannot be relinked.
pii_redaction:
enabled: false
# redact_email: true
# redact_api_key: true
# redact_credit_card: true
# redact_phone: true
# redact_national_id: true
# ============================================================================
# Tool Progress State Machine Configuration (RFC #3177)
# ============================================================================