From 2ff006b0c09ad3a7e48515018a4ddad7af45ba59 Mon Sep 17 00:00:00 2001 From: xiaodu55 <1704788159@qq.com> Date: Sat, 19 Sep 2026 11:26:46 +0800 Subject: [PATCH] feat(middleware): add deterministic PII redaction for model-bound context (#5527) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(middleware): add deterministic PII redaction for model-bound context * fix(middleware): claim national IDs before cards, redact Command results, preserve ToolMessage fields - Reorder detectors so checksum-gated national IDs run before the credit-card detector; an 18-digit resident ID whose digit run also passes Luhn is no longer mislabeled [CREDIT_CARD_n] (review finding, reproduced at 0a2a9d0) - Redact ToolMessages carried in Command.update.messages, mirroring ToolResultSanitizationMiddleware's dc_replace pattern - Rebuild redacted ToolMessages via model_copy so artifact and response_metadata survive - Extend the numbered middleware chain in agents/middlewares/AGENTS.md * fix(middleware): span one redactor per Command result; refresh stale AGENTS.md entry range - Placeholder numbering now continues across every ToolMessage carried in a single Command result (one _Redactor per _redact_result call) instead of restarting per message - The renumbered AGENTS.md chain still referenced entries 9-12 in the ToolReceiptMiddleware entry; it now reads entries 10-13 * docs(agents): trim PiiRedactionMiddleware entry to fit the AGENTS.md chain budget The main merge (fb36e0e) pushed the effective middlewares chain to 98341 bytes, 37 over the 98304 hard limit checked by agent-guidance (AG002). Compress the entry while keeping the load-bearing facts: config gate, both interception points incl. Command coverage, detector order rationale, per-result numbering continuity, irreversibility, memory follow-up. * fix(middleware): redact compaction input and reinjected summaries; harden detectors Review round 3 on #5527: - [P1] SummarizationMiddleware invokes its summary model directly from before_model, outside PiiRedactionMiddleware's wrap_model_call, so raw thread state reached the summary model and reinjected summaries carried raw PII into model-bound context. Add a shared redact_text() seam: the compaction prompt is redacted in _build_summary_prompt (app_config already flows into the middleware) and DurableContextMiddleware redacts summary_text at reinjection via a new pii_redaction_config knob wired at both assembly sites. - [P2] CUIT is 2+8+1 digits, not 2+10+1. - [P2] Digit-anchored patterns use digit-aware lookarounds instead of Unicode \b, which CJK characters defeat (身份证110105… / 手机号138…). - [P2] The international-phone pattern no longer treats newlines as separators, so a candidate cannot swallow the following numeric field and then fail validation as a whole. * fix(pii): redact title input and reserve summary placeholders --------- Co-authored-by: Willem Jiang --- README.md | 9 + .../deerflow/agents/lead_agent/agent.py | 1 + .../deerflow/agents/middlewares/AGENTS.md | 68 +- .../middlewares/durable_context_middleware.py | 7 +- .../middlewares/pii_redaction_middleware.py | 463 ++++++++++++ .../middlewares/summarization_middleware.py | 6 + .../agents/middlewares/title_middleware.py | 8 +- .../tool_error_handling_middleware.py | 10 + .../harness/deerflow/config/app_config.py | 2 + .../deerflow/config/pii_redaction_config.py | 39 + .../tests/test_pii_redaction_middleware.py | 666 ++++++++++++++++++ config.example.yaml | 20 + 12 files changed, 1263 insertions(+), 36 deletions(-) create mode 100644 backend/packages/harness/deerflow/agents/middlewares/pii_redaction_middleware.py create mode 100644 backend/packages/harness/deerflow/config/pii_redaction_config.py create mode 100644 backend/tests/test_pii_redaction_middleware.py diff --git a/README.md b/README.md index 2155e7875..5b407c2c0 100644 --- a/README.md +++ b/README.md @@ -1402,6 +1402,15 @@ The Web UI shows the active goal above the composer. The same command is availab ### Manual Context Compaction +Optional `pii_redaction.enabled` redacts detected identifiers in user messages, +remote tool results, compaction input, reinjected summaries, and configured +LLM title input. It is off by default. Existing summary placeholders reserve +indices so new values do not reuse them after compaction. No PII mapping is +persisted, so repeated values cannot be linked to a compacted source; numbering +may change when history or summary placeholders disappear. Raw thread text and +local fallback titles remain available for display; memory extraction is outside +this feature's scope. + The Web UI preserves persisted message order when merging history with live updates. Streaming steps around a persisted result inside the loaded history stay together, including steps that arrive after the result. Steps captured during compaction also remain visible before their persisted result when history has not refreshed and the UI has not rendered them yet. Compaction keeps the current user request and summarizes older assistant/tool activity. When rescuing that request leaves an assistant/tool-only summary window, input trimming favors its most recent content. For mixed histories whose user-message anchor falls outside the trimming budget, compaction retains the existing final-message fallback. `summarization.trim_tokens_to_summarize` (4000 by default) controls trimming of the raw summary input; escaping and prompt formatting add overhead beyond that budget. Setting this option to `null` disables input trimming for the summary model; choose that only when the model can accept the full history being compacted. diff --git a/backend/packages/harness/deerflow/agents/lead_agent/agent.py b/backend/packages/harness/deerflow/agents/lead_agent/agent.py index b2620bd5b..a03c9b2d1 100644 --- a/backend/packages/harness/deerflow/agents/lead_agent/agent.py +++ b/backend/packages/harness/deerflow/agents/lead_agent/agent.py @@ -608,6 +608,7 @@ def build_middlewares( skills_container_path=resolved_app_config.skills.container_path, skill_file_read_tool_names=resolved_app_config.summarization.skill_file_read_tool_names, task_continuity_enabled=getattr(getattr(resolved_app_config, "task_continuity", None), "enabled", False) is True, + pii_redaction_config=getattr(resolved_app_config, "pii_redaction", None), ) ) diff --git a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md index a978a9ac7..f1a9747ec 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md +++ b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md @@ -69,16 +69,18 @@ strict providers reject. ordered by application — the last entry produced the final visible bytes — so an observer classifies raw→visible transforms from facts rather than by sniffing output wording. -4. **ThreadDataMiddleware** - Creates per-thread directories under the user's isolation scope (`backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); resolves identity via `resolve_runtime_user_id(runtime)`, including Gateway runtime context and standalone LangGraph Server auth, then falls back to the request ContextVar / `"default"` -5. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation (lead agent only); upload existence checks use the same runtime-resolved user bucket as thread-data creation -6. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state. The +4. **PiiRedactionMiddleware** - *(optional, `pii_redaction.enabled`, default off, #3190)* Rewrites PII in genuine user messages (`wrap_model_call`, request-scoped, raw text stays in thread state) and remote-content tool results (`wrap_tool_call`, ToolResultSanitizationMiddleware's allowlist incl. `Command.update.messages`) to irreversible placeholders (`[EMAIL_1]`, …). Deterministic regex detectors only, no new dependencies; national IDs run before the card detector so a checksum-valid resident ID whose digits also pass Luhn is never mislabeled `[CREDIT_CARD_n]`. One redactor per result keeps numbering continuous; existing summary/message tokens reserve indices before new values; no persistent mapping links compacted identities. Innermost Layer-1 wrapper, so budget-externalized copies hold redacted text; compaction, reinjected summaries, and title-model inputs are redacted; title fields are redacted before truncation. Memory extraction is a follow-up slice. + +5. **ThreadDataMiddleware** - Creates per-thread directories under the user's isolation scope (`backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); resolves identity via `resolve_runtime_user_id(runtime)`, including Gateway runtime context and standalone LangGraph Server auth, then falls back to the request ContextVar / `"default"` +6. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation (lead agent only); upload existence checks use the same runtime-resolved user bucket as thread-data creation +7. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state. The lead runtime normally owns the thread's physical Agent-skill projection; delegated subagents and the prompt-only bootstrap agent are non-owners, so their narrower discovery allowlists never rebuild the shared thread view or force eager sandbox acquisition. -7. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., user interruption), preserving raw provider tool-call payloads in `additional_kwargs["tool_calls"]`; malformed tool-call names and arguments are sanitized in the model-bound request so strict OpenAI-compatible providers do not reject the next request -8. **LLMErrorHandlingMiddleware** - Converts provider/model failures to recoverable assistant errors. Normal completions without visible text or tool-call intent (including whitespace/reasoning-only responses) get at most one retry per run, then a marked visible fallback; empties never count toward the circuit breaker. Cancellation during admission, execution, retry events or backoff releases only this call's half-open probe (assigned under the circuit lock), then propagates unchanged without retry or failure accounting. -9. **Authorization / GuardrailMiddleware** - Up to two independent pre-tool-call gates run here. When `authorization.enabled`, the `AuthorizationProvider` instance already used for Layer 1 capability filtering is wrapped by `GuardrailAuthorizationAdapter` and reused for Layer 2 execution checks. A generated `tool_search` bypasses the adapter's second provider call only when the current build has a concrete deferred setup; its catalog was already filtered by Layer 1, and an ordinary same-named tool without that deferred setup receives no exemption. When `guardrails.enabled`, the explicitly configured `GuardrailProvider` is appended after authorization and still evaluates every call, including `tool_search`. Authorization therefore runs outermost and can deny before an external guardrail call; both use the existing middleware's fail-closed, audit, sync/async, and error-`ToolMessage` behavior. See the authorization RFC and [docs/GUARDRAILS.md](../../../../../docs/GUARDRAILS.md). +8. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., user interruption), preserving raw provider tool-call payloads in `additional_kwargs["tool_calls"]`; malformed tool-call names and arguments are sanitized in the model-bound request so strict OpenAI-compatible providers do not reject the next request +9. **LLMErrorHandlingMiddleware** - Converts provider/model failures to recoverable assistant errors. Async cancellation at admission, provider execution, retry events, or backoff releases only the call's own half-open probe (ownership assigned under the circuit lock), then propagates unchanged, without retry or failure accounting. +10. **Authorization / GuardrailMiddleware** - Up to two independent pre-tool-call gates run here. When `authorization.enabled`, the `AuthorizationProvider` instance already used for Layer 1 capability filtering is wrapped by `GuardrailAuthorizationAdapter` and reused for Layer 2 execution checks. A generated `tool_search` bypasses the adapter's second provider call only when the current build has a concrete deferred setup; its catalog was already filtered by Layer 1, and an ordinary same-named tool without that deferred setup receives no exemption. When `guardrails.enabled`, the explicitly configured `GuardrailProvider` is appended after authorization and still evaluates every call, including `tool_search`. Authorization therefore runs outermost and can deny before an external guardrail call; both use the existing middleware's fail-closed, audit, sync/async, and error-`ToolMessage` behavior. See the authorization RFC and [docs/GUARDRAILS.md](../../../../../docs/GUARDRAILS.md). Every guardrail decision path publishes a neutral `deerflow.authz.outcome.AuthorizationOutcome` into the per-run runtime context, @@ -86,10 +88,10 @@ strict providers reject. `__authorization_outcome` key (so `build_run_config` strips caller-supplied forgeries). Consumers pop it; the publisher and the consumer share only that contract module. -10. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations before tool execution; command classification is **defense-in-depth and audit, not a security boundary** (the sandbox is the isolation boundary). Command substitution is judged by *position*, not the presence of `$(`: **command position** (`$(curl url)`, `` `curl url` ``, the word after `|`/`&&`/`;`, an `eval`/`source` argument) executes fetched content and is blocked; **value position** (`x=$(curl url)`, `echo $(curl url)`, an argument, a `for` word list) only captures output and passes (#4611). So `_HIGH_RISK_COMMAND_POSITION_PATTERNS` is matched anchored against each sub-command from `_split_compound_command(split_pipes=True)`, never the whole string; pipe-spanning rules (`| sh`, `base64 -d | ...`) still use `_classify_command`'s whole-command Pass 1. `_COMMAND_POSITION_PREFIX` extends the anchor over leading assignments and exec wrappers (`FOO=1 $(curl url)`, `env`/`command`/`builtin`/`exec`/`nohup`/`time`/`sudo`/`doas`); its assignment branch requires whitespace before the substitution, which keeps `x=$(curl url)` in value position. Two contexts are deliberately **position-blind** (matched whole-command in Pass 1, since they execute their input anywhere, e.g. `xargs sh -c "$(curl url)"`): an `eval`/`source` argument, and an interpreter **code-string flag** — `-c` (shells, `python`), `-e` (`perl`/`ruby`/`node`), `-p` (`perl`/`node`), `-r` (`php`) — plus the here-string (`<<<`) reaching the same place via stdin. All three substitution spellings (`$(`, `<(`, `` ` ``) share one `_RISKY_SUBSTITUTION` opener. An unquoted newline splits like `;` (else `echo hi\n$(curl url)` evades the anchored rules). Heredoc bodies are data: `_split_compound_command` records headers (`<