mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-24 21:46:17 +00:00
2ff006b0c0
* feat(middleware): add deterministic PII redaction for model-bound context * fix(middleware): claim national IDs before cards, redact Command results, preserve ToolMessage fields - Reorder detectors so checksum-gated national IDs run before the credit-card detector; an 18-digit resident ID whose digit run also passes Luhn is no longer mislabeled [CREDIT_CARD_n] (review finding, reproduced at 0a2a9d0) - Redact ToolMessages carried in Command.update.messages, mirroring ToolResultSanitizationMiddleware's dc_replace pattern - Rebuild redacted ToolMessages via model_copy so artifact and response_metadata survive - Extend the numbered middleware chain in agents/middlewares/AGENTS.md * fix(middleware): span one redactor per Command result; refresh stale AGENTS.md entry range - Placeholder numbering now continues across every ToolMessage carried in a single Command result (one _Redactor per _redact_result call) instead of restarting per message - The renumbered AGENTS.md chain still referenced entries 9-12 in the ToolReceiptMiddleware entry; it now reads entries 10-13 * docs(agents): trim PiiRedactionMiddleware entry to fit the AGENTS.md chain budget The main merge (fb36e0e) pushed the effective middlewares chain to 98341 bytes, 37 over the 98304 hard limit checked by agent-guidance (AG002). Compress the entry while keeping the load-bearing facts: config gate, both interception points incl. Command coverage, detector order rationale, per-result numbering continuity, irreversibility, memory follow-up. * fix(middleware): redact compaction input and reinjected summaries; harden detectors Review round 3 on #5527: - [P1] SummarizationMiddleware invokes its summary model directly from before_model, outside PiiRedactionMiddleware's wrap_model_call, so raw thread state reached the summary model and reinjected summaries carried raw PII into model-bound context. Add a shared redact_text() seam: the compaction prompt is redacted in _build_summary_prompt (app_config already flows into the middleware) and DurableContextMiddleware redacts summary_text at reinjection via a new pii_redaction_config knob wired at both assembly sites. - [P2] CUIT is 2+8+1 digits, not 2+10+1. - [P2] Digit-anchored patterns use digit-aware lookarounds instead of Unicode \b, which CJK characters defeat (身份证110105… / 手机号138…). - [P2] The international-phone pattern no longer treats newlines as separators, so a candidate cannot swallow the following numeric field and then fail validation as a whole. * fix(pii): redact title input and reserve summary placeholders --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
40 lines
1.5 KiB
Python
40 lines
1.5 KiB
Python
"""Configuration for the PII redaction middleware (issue #3190)."""
|
|
|
|
from pydantic import BaseModel, Field
|
|
|
|
|
|
class PiiRedactionConfig(BaseModel):
|
|
"""Configuration for deterministic PII redaction in model-bound context.
|
|
|
|
Default-off. When enabled, personally identifiable information found in
|
|
genuine user messages and remote-content tool results is rewritten to
|
|
irreversible placeholders (``[EMAIL_1]`` …) before it reaches the model.
|
|
Each detector can be toggled independently for deployments that only need
|
|
a subset (e.g. credentials but not phone numbers).
|
|
"""
|
|
|
|
enabled: bool = Field(
|
|
default=False,
|
|
description="Whether to enable PII redaction in model-bound context",
|
|
)
|
|
redact_email: bool = Field(
|
|
default=True,
|
|
description="Redact email addresses",
|
|
)
|
|
redact_api_key: bool = Field(
|
|
default=True,
|
|
description="Redact API keys and bearer-style tokens (OpenAI sk-, AWS AKIA, GitHub ghp_/github_pat_, Slack xox-, Google AIza)",
|
|
)
|
|
redact_credit_card: bool = Field(
|
|
default=True,
|
|
description="Redact credit-card numbers passing the Luhn checksum",
|
|
)
|
|
redact_phone: bool = Field(
|
|
default=True,
|
|
description="Redact phone numbers (international +CC form, CN mobile, US formatted)",
|
|
)
|
|
redact_national_id: bool = Field(
|
|
default=True,
|
|
description="Redact national IDs (CN resident ID and CPF with checksum validation, formatted CUIT/RFC)",
|
|
)
|