* fix(mcp): reject credentials that cannot travel as HTTP header values
A request-scoped secret or user_auth credential with a trailing newline
(the usual result of reading a token from a file, or a CRLF env-file),
CR/LF, surrounding whitespace, or characters outside Latin-1 sailed
through the credential interceptors into the HTTP client, where httpx/h11
reject it with an exception that echoes the full value:
LocalProtocolError: Illegal header value b'Bearer sk-...\n'
ToolErrorHandlingMiddleware copies that message into a model-visible
ToolMessage, so the secret landed in the prompt, the checkpoint, and
traces - everywhere headers_from_context promises it never goes.
Add illegal_header_value_reason to mcp/headers.py, mirroring the
transport's own rules (Latin-1 encodable; h11's field_vchar is [^\x00\s]
with SP/HTAB legal only between visible characters), and fail closed in
both interceptors before the value can reach the client. The denial names
only the secret key (plus the reason) and never repeats the value.
Illegal values are denied regardless of on_missing: the key is present,
so a passthrough fallback would silently run the call under the shared
discovery credential - the exact authority confusion the deny default
exists to prevent.
Values the transport accepts are not rejected: embedded SP/HTAB
('Bearer <token>'), Latin-1 high bytes, and DEL all still pass, pinned
by tests against h11's observed behaviour.
* fix(mcp): tighten header value validation to httpx's ASCII boundary
The validator mirrored h11's Latin-1 boundary, but the transport rejects
more than h11 does: build_server_params hands dict[str, str] headers
through the MCP SDK's create_mcp_http_client into httpx.AsyncClient, and
httpx (pinned 0.28.1) encodes str header values as ASCII - so a Latin-1
high byte like 'Bearer caf\xe9' passed validation here only to raise
UnicodeEncodeError inside httpx before h11 ever ran, with the exception
message repeating the offending value.
Validate str values against ASCII instead, flip the tests that pinned
Latin-1 high bytes as transportable, and pin the boundary against the
real client: create_mcp_http_client must reject what the validator
flags and construct cleanly for what it accepts (embedded SP/HTAB and
DEL still pass).
Addresses review feedback on the ASCII vs Latin-1 boundary.
* fix(mcp): validate OAuth and static header values at the same boundary
The validator added for headers_from_context and user_auth left two paths
uncovered. A token endpoint returning an access_token or token_type with a
newline reached httpx/h11, which raise with the full token in the message, and
ToolErrorHandlingMiddleware copies that message into a model-visible
ToolMessage -- the leak this PR set out to close. The operator's static headers
had the same hole.
OAuthTokenManager.get_authorization_header now renders the Authorization value
through one checked helper, so the tool interceptor, the initial discovery
headers and the durable task path are all covered by a single guard. The
rendered value is what gets checked rather than the two fields separately,
because that is what the transport sees: an access_token with leading
whitespace is legal once it follows "Bearer ".
build_server_params applies the same check to statically configured headers.
build_servers_config already isolates a per-server failure, so a bad value
drops that one server and logs the reason instead of the value.
* docs(mcp): correct which transport echoes the full header value
The rationale claimed httpx and h11 both render the full value into their
exception message. Only h11 does, on the line break and surrounding whitespace
cases. httpx's ASCII failure is a UnicodeEncodeError naming the offending
character and its position, not the credential, so at most one character
escapes there; refusing the value up front buys an actionable error rather than
an encode failure raised from inside the client.
Corrected in headers.py and in every copy of the claim: context_headers.py,
user_scoped_auth.py, oauth.py, client.py, mcp/AGENTS.md, docs/MCP_SERVER.md,
the frontend mcp.mdx, and the test comments carrying the same wording. No
behavior change.
---------
Co-authored-by: Terminator666666 <Terminator666666@users.noreply.github.com>
Add deerflow.community.serply.tools:web_search_tool, a Google SERP
provider for the web_search slot that also covers Google News and Google
Scholar through an optional `vertical` config option. Reads the key from
api_key in config.yaml or SERPLY_API_KEY, clamps max_results to Serply's
1-100 range, and returns the same structured JSON errors as the Serper
and Brave tools.
Register the provider in config.example.yaml, scripts/doctor.py,
scripts/wizard/providers.py, .env.example, backend/docs/CONFIGURATION.md,
the en/zh tools.mdx provider tabs, and tools/AGENTS.md. Tests mock httpx.
* feat(mcp): map request-scoped secrets to HTTP/SSE headers
`user_auth` binds a credential to a configured DeerFlow user, so a caller
that picks the credential per request — a multi-tenant gateway, a per-run
API key, one shared MCP server fronting several environments — had to
register one MCP server entry per credential.
Add a declarative `mcpServers.<server>.headers_from_context` block mapping
HTTP header names to keys of the run request's `config.context.secrets`
carrier. A new built-in interceptor resolves the mapping on every tool call
and rewrites those headers, mirroring `user_scoped_auth`. The config file
stores names only, never a credential, so the Gateway returns the block
unmasked.
Registered after OAuth and `user_auth` in the interceptor chain: the later
interceptor runs closer to the transport, and the value chosen for this one
request is the most specific, so it wins. Fail-closed by default — a mapped
key missing from the request raises a `ToolException` naming only that key,
because falling back to the server's discovery credential would send one
tenant's call under another tenant's authority. `on_missing: "passthrough"`
opts out.
Durable background tasks are excluded: `McpTaskToolCaller` drives status and
cancel polls after the Agent run ends, where no run context exists, so the
fail-closed interceptor would deny every poll. Those calls keep using
server-level credentials, and a server declaring both `headers_from_context`
and `task_toolsets` now logs a warning.
Also corrects the custom-interceptor example in docs/MCP_SERVER.md (and the
matching claim in skills/AGENTS.md), which read request secrets from
`langgraph.config.get_config()["context"]`. That key is `None` inside a tool
call — the run context rides the LangGraph runtime, not the RunnableConfig
propagated to child runnables — so interceptors written from that example
never saw a value. The example now reads `request.runtime`, and
tests/test_mcp_context_headers.py pins LangGraph's runtime-injection rule by
driving a real langchain-mcp-adapters tool through a real graph with the
ambient-runtime fallback disabled.
Closes#5005
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(mcp): resolve credential headers case-insensitively, carry them on durable submit
Review follow-ups on `headers_from_context`.
HTTP field names are case-insensitive, but every dict on the path to the wire
is not: `build_server_params` copies the operator's static `headers` spelling
verbatim, and langchain-mcp-adapters merges interceptor overrides into the
connection with a plain `{**connection_headers, **override_headers}` splat. A
static `authorization` and an injected `Authorization` therefore both reached
httpx as separate field lines, and a server reading the field with a
single-value accessor got the static discovery credential — inverting the
documented `headers` < `oauth` < `user_auth` < `headers_from_context`
precedence and running a per-request call under the shared credential.
Normalizing inside the interceptor cannot fix that on its own: the adapter
builds the request with `headers=None`, so an interceptor never sees the
connection's static headers and cannot displace them however it spells its own
key. A new `mcp/headers.py::apply_header_overrides` therefore drops any key
differing only in case and emits the spelling the connection already uses.
Applied to `headers_from_context`, `user_auth`, the OAuth interceptor, the
OAuth discovery-header write, and the durable-task connection merge, which all
carried the same collision. `headers_from_context.headers` now also rejects one
header mapped under two spellings at config load, in both the harness model and
the Gateway mirror.
Durable submit now carries the mapped headers, as docs/MCP_SERVER.md already
promised. `McpTaskToolCaller` disabled the interceptor for the whole caller, but
that caller serves submit as well as the polls, and submit is awaited inline
inside the Agent's tool call — where the run's LangGraph runtime is still the
ambient contextvar, so no secret has to be threaded through `TaskSubmitRequest`
or reach durable storage. The caller builds one chain and keeps a second view of
it without the context-headers interceptor; `call_tool` takes
`request_scoped_headers`, set only by `OrdinaryMcpTaskDriver.submit`. Status and
cancel keep server-level credentials, so background polls still cannot fail
closed, and the startup warning now describes the half it actually covers.
`_merge_preserving_secrets` restores masked extras inside `headers_from_context`
instead of writing the `***` sentinel back over the stored value, matching the
treatment `user_auth` extras and server-level extras already get; extras a PUT
omits carry over as well, while the declared mapping still replaces verbatim so
a round trip can remove an entry. `extra="allow"` plus name-based sensitivity
detection means the usual casualty is a name-valued key such as `tokenHeader`,
not only a credential.
The existing override test seeded the static header onto `request.headers`,
which production never does, so it modelled a merge that really happens one
layer down; the new tests drive a real adapter tool through a real connection
and assert on the headers the session is opened with, and the durable-submit
test runs through a real tool node with no runtime patching.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(mcp): reject case-insensitive duplicate static header names
* fix(mcp): preserve omitted headers_from_context fields on partial updates
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat(mcp): per-user credential injection for shared MCP servers
A single HTTP/SSE MCP server entry can now serve several users, each
authenticated to the remote service with their own credential. A server
opts in with a user_auth block mapping user ids to credential header
values ($ENV_VAR references supported):
"user_auth": {
"header": "Authorization",
"users": { "<user-id>": "$SERVICE_TOKEN_ALICE" }
}
The built-in user-scoped auth interceptor resolves the authenticated
runtime user on every tool call (request runtime -> ambient LangGraph
runtime -> auth config -> request-scoped user ContextVar) and rewrites
the configured header via request.override(), the same per-call
mechanism as the OAuth interceptor. It registers after OAuth in the
shared assembly so its per-user value wins the header when a server
declares both. The entry's static headers are used only for startup
tool discovery.
Fail-closed by default: an unmapped user - including the anonymous
default-user fallback - or a credential whose env reference resolved
empty gets an actionable ToolException instead of another user's
credential; on_missing: "passthrough" opts out per server. Combined
with the existing per-(user, thread) MCP session scoping this gives
credential isolation on shared servers.
Gateway API: user_auth.users values are masked in GET responses, and
PUT round-trips preserve stored credentials for masked values (same
contract as env/headers/oauth secrets); a masked value for a user id
not already stored is rejected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): address review — preserve stored user_auth sub-fields on partial PUT, warn-and-skip stdio, allow extras
Review findings from #4868:
1. A partial user_auth payload (e.g. {"enabled": false}) merged to
users={} and default on_missing, irreversibly wiping stored
credentials on PUT. The merge is now sub-field-aware via
model_fields_set — omitted sub-fields carry the stored values, an
explicitly sent users map still replaces (so full-round-trip removal
works), masked values still swap back for stored credentials.
2. user_auth on a stdio server was a silent no-op: rewritten headers go
to call meta, never a transport header, while deny errors still fired.
The interceptor builder now warns and skips non-sse/http servers,
matching the tool_call_timeout transport-mismatch convention.
3. McpUserScopedAuthConfigResponse now allows extra keys like the
harness-side model, and extras survive masking and merge, matching
the server-level model_extra handling.
Adds four regression tests (partial-PUT preservation, explicit-map
replacement, extras round-trip, stdio warn-and-skip).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): reject blank user_auth.header at the gateway
A blank header passed the gateway response model, was persisted, then
failed the harness-side ExtensionsConfig validator on reload — the PUT
returned 500 after the write and every later config load/startup failed
until the file was hand-edited. Mirror the harness non-blank validator
on McpUserScopedAuthConfigResponse so the PUT fails with 422 before
anything is written.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: ruff format extensions_config.py
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): harden the trust chain and masking around user-scoped credential selection
Address review round 4:
- Scrub client-supplied user_id from run context/configurable for
external callers in inject_authenticated_user_context, before every
early return, and restamp only from request.state.user. Now that
user_id selects which user's credential user-scoped MCP auth injects,
a forged value must not survive any future path that skips the
restamp. Internal callers (IM channels, scheduler) keep supplying
end-user identity as before (PR #3294 contract). Regression tests pin
both the scrub and that a forged body.context.user_id can never
resolve as another user through merge + inject ordering.
- Include the caller's own resolved user id in the fail-closed deny
message so operators can copy the exact users key (it differs by
deployment path), and document the key formats in the mcp.mdx doc.
- Mask sensitive extra keys inside user_auth on GET like server-level
extras, and swap masked sentinels back for stored values on PUT via
_merge_extra_value_preserving_masked.
- Extract the interceptor wrap loop into compose_tool_interceptors and
pin the security property functionally: an OAuth interceptor that
actually sets Authorization loses the final header to the per-user
credential through the same composition the session-pool path uses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sync the harness docs with the #3875 subagent middleware changes:
- subagents.mdx (en/zh): fix built-in max_turns defaults (general-purpose
160->150, bash 80->60) to match code and config.example.yaml; document the
new config (global + per-agent override) and add a
'Runaway guards' section covering the LoopDetection / TokenBudget /
Summarization middlewares now mirrored on the subagent chain.
- middlewares.mdx (en/zh): note that loop-detection, token-budget, and
summarization guards are shared with the subagent chain (#3875), instead of
implying they are Lead-Agent-only.
Code, contracts/subagent_status_contract.json (v2), and config.example.yaml
already reflect #3875; only the docs were lagging.
* feat(memory): pluggable + self-contained memory system (MemoryManager plan phases 1 & 2)
Phase 1 — Pluggable (steps 0-10):
- ABC MemoryManager (9 methods) + singleton factory + drop-in backend discovery
- DeerMem default backend with core/ (storage/queue/updater/prompt/message_processing)
- NoopMemoryManager backend (proves pluggability)
- All call sites (middleware/hook/prompt/gateway/client/app) routed through manager
- hasattr capability probing for DeerMem-internal methods (no hard imports)
- MemoryConfig gains manager_class field; shared vs DeerMem-private annotated
Phase 2 — Self-contained DeerMem (steps 11-18):
- backend_config passthrough + DeerMemConfig (all DeerMem-private fields moved off MemoryConfig)
- DI: DeerMem owns storage/queue/updater/llm as instance attributes (no global singletons)
- Storage independence: core/paths.py with own root (~/.deermem or ),
factory auto-injects deer-flow's runtime_home() as absolute base_dir (zero-config)
- LLM independence: core/llm.py via langchain init_chat_model (no create_chat_model)
- Trace independence: optional tracing_callback replaces inject_langfuse_metadata/request_trace_context
- Message processing independence: hide_from_ui default-skip + optional should_keep_hidden_message hook
- Internal imports → relative (only deer_mem.py ABC import is host-relative)
- Carrier (deer_mem.py adapter) / portable (deermem/ config+core) split
- New tests: test_deermem_self_contained + test_memory_manager_pluggable; all memory tests migrated
- Other-agent demo: samples/other_agent_demo/ + automated portability test
- config.example.yaml memory section updated to phase-2 schema
* feat(memory): port consolidation + staleness fix into self-contained DeerMem; phase-2 host hooks
Port upstream #3996 (memory consolidation) and #3993 (staleness KeyError fix)
from origin/MemoryManager into the pluggable, self-contained DeerMem structure
(backends/deermem/deermem/), adapted to the DI MemoryUpdater (config injected,
not get_memory_config globals):
- DeerMemConfig: add consolidation_enabled (opt-in, default false) /
consolidation_min_facts / consolidation_max_groups_per_cycle /
consolidation_max_sources
- prompt.py: factsToConsolidate JSON field + {consolidation_section} placeholder
+ CONSOLIDATION_PROMPT constant
- updater.py: _coerce_source_confidence / _select_consolidation_candidates /
_build_consolidation_section module helpers (matching the existing
_select_stale_candidates style); consolidation normalization in
_normalize_memory_update_data; consolidation apply in _apply_updates (after
max_facts trim, with apply-time guardrails mirroring staleness); staleness
KeyError fix (f["id"] -> f.get("id") is not None) applied to both the
staleness guardrail and the consolidation allowed_source_ids comprehension
- config.example.yaml: consolidation section under memory.backend_config
- tests/test_memory_consolidation.py: 40 DI-adapted tests (running, not skipped)
incl. the staleness KeyError regression
Also includes in-flight phase-2 host-integration work: storage_path semantics
(any absolute/relative value = root dir) and host-default tracing_callback /
should_keep_hidden_message hooks injected into backend_config by the factory.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(memory): add noop backend template and backends guide
- backends/noop/: complete drop-in template (config.py with zero deer-flow
imports, noop_manager.py with a 6-step new-backend walkthrough in its
docstring, commented optional fact-CRUD capabilities).
- backends/README.md: which files to touch when adding/swapping a backend,
the 5-item backend contract, and common pitfalls.
- manager.py: generalize backend examples in comments (drop mem0-specific
references).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): guard formatTimeAgo against invalid timestamps
Return a neutral placeholder when the input date is invalid (e.g. an empty lastUpdated from a backend with no memories) instead of throwing 'Invalid time value' from date-fns.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(memory): wire tool-driven memory mode through the MemoryManager ABC
tools.py (memory_search/add/update/delete) now calls get_memory_manager()
instead of the removed host memory module, so tool mode (memory.mode: tool)
works for any backend. DeerMem.search is implemented (case-insensitive
substring match, ranked by confidence) as a stand-in for the planned
semantic retrieval; noop.search returns [] (unchanged). Fact-CRUD tools
use getattr+callable probing -- backends lacking those ops (noop) get a
clear JSON error instead of crashing.
Tests: test_memory_tools rewired to mock the manager (handler tests) +
TestModeGating retained; test_memory_search now covers DeerMem.search;
pluggable stubs test updated (search no longer a stub).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: resolve lint errors (import sorting, type annotation quotes, E402 in skipped tests)
* docs: restore explanatory comments in config.example.yaml memory section
* fix(security): port html-escape memory facts fix (#4097) to vendored DeerMem prompt.py
* fix(memory): address review + port dropped upstream memory fixes
Review blockers (vendored DeerMem):
- #4044 restore _escape_memory_for_prompt (current_memory blob in
MEMORY_UPDATE_PROMPT) - prevents </current_memory> breakout
- #4028 html.escape staleness-section cat/content in _build_staleness_section
- #4119 add _escape_summary for injection-path summaries (Work/Personal/
Current Focus/Recent/Earlier/Background)
- default-model silent no-op: factory injects host default chat model via a
new host_llm slot (create_chat_model(name=None)); DeerMem prefers host_llm
over build_llm(model). Zero-config extraction works out of the box again
- MemoryConfigResponse: fix stale docstring (backend-agnostic shape; DeerMem
knobs live under backend_config, not top-level - restoring flat would
re-couple the API to DeerMem). Frontend audited: does not read /memory/config
- _host_default_tracing_callback: restore langfuse assistant_id/environment
- search: push category onto the ABC signature; DeerMem filters BEFORE the
top_k slice (was filtered client-side after slicing -> starved results)
- _do_update_memory_sync: split into wrapper+impl; bind trace_id into the
request-trace ContextVar on the Timer/executor worker via a new
trace_context_manager host hook (None trace_id left unbound - no fabrication)
- client.py fact-CRUD now passes user_id (was writing to the global bucket
while get_memory reads per-user)
- _resolve_manager_class: fail-fast (raise ValueError) on an unresolved
explicit manager_class instead of silently falling back to DeerMem (memory is
persistent state - a wrong store is a silent data-integrity footgun)
Upstream memory fixes dropped by the host->vendored rename conflict, re-ported
to backends/deermem/deermem/core/ (+ deer_mem.py):
- #4073 queue busy-timer-spin -> _reprocess_pending flag (core/queue.py)
- #4074 null source.confidence in staleness -> _coerce_source_confidence
(core/updater.py: _build_staleness_section + _apply_updates stale sort)
- #4075 factsToRemove is optional (drop from _REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS)
- #4076 null confidence in search ranking -> _coerce_source_confidence
(deer_mem.py DeerMem.search)
host_llm + trace_context_manager are host-injected via backend_config (factory
in manager.py), keeping backends/deermem/ at exactly one `from deerflow` line
(the ABC contract) - portability test preserved.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: resolve lint errors (F541 f-string without placeholders, E501 line too long)
* fix(memory): restore hide_from_ui clarification preservation, expose mode
Two memory-system fixes (F541/E501 lint was already fixed on this branch):
- filter_messages_for_memory: restore default preservation of well-formed
human_input_response clarification answers (v2 regression). The
self-containment refactor made the bare function skip ALL hide_from_ui when
no hook was passed, but upstream preserves well-formed clarification
responses by default (test_hide_from_ui_human_input_response_is_preserved).
Inline a host-agnostic _is_human_clarification_response mirror of
read_human_input_response as the default keep-decision; the host-injected
should_keep_hidden_message hook still overrides (production path unchanged).
Portable package stays zero `from deerflow`.
- /memory/config: expose `mode` (middleware|tool) in MemoryConfigResponse +
the config/status endpoints + client.get_memory_config. mode is a host-
shared, behavior-determining field missing from the response projection.
Sync tests (mock .mode; e2e assert mode present).
- Align manager_class field docstring with fail-fast behavior.
Tests: filter/self-contained/portability (35) + memory-config (4) pass;
ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): resolve ruff format failures in memory module + tests
`make lint` runs `ruff format --check` in addition to `ruff check`; 8 memory
files had pending format changes -- 7 pre-existing (deer_mem, updater, tools,
test_memory_queue/router/search/tools) + message_processing from the
hide_from_ui fix. Apply `ruff format`: whitespace/wrapping only, no logic
change. 109 memory tests pass; ruff check + format --check both clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): address PR review - legacy field migration, fact_id contract, path/docs
Address willem-bd's review on PR head bc8bf0d4 (risk:high, persistent state):
- config: auto-migrate pre-abstraction top-level memory.* DeerMem fields
(storage_path, max_facts, debounce_seconds, model_name, token_counting,
staleness_*, consolidation_*) into backend_config on load + warn, so an
upgrade does NOT silently revert customized settings (was: silent
extra='ignore' drop). model_name -> backend_config.model.model. Unknown
top-level keys warned.
- factory: resolve a relative backend_config.storage_path against runtime_home()
(base_dir-relative, CWD-independent) to preserve pre-abstraction semantics;
paths.py stays portable (no runtime_home import).
- tools: memory_add uses the fact_id returned directly by create_fact instead of
re-deriving it via content-key matching (coupled the tool to the backend's
content normalization; could misreport a storage cap). create_fact now returns
(memory_data, fact_id); gateway/client/tool updated. Fix terse
{"error":"content"} -> {"error":"empty content"}.
- app.py: update stale token_counting=="char" warm-up comment to point at
manager.warm (DeerMem.warm re-checks char and returns early).
- router: comment explaining reload_memory silent fallback vs fact 501 asymmetry
(read-only degrade vs write fail-loud).
- CHANGELOG: document breaking changes (/memory/config + client.get_memory_config
shape flat->backend_config; custom storage_class path moved + __init__ must
accept config) and the legacy-field auto-migration.
- tests: add regression test pinning the per-user memory path
({storage_path}/users/{safe_user_id}/memory.json == host make_safe_user_id)
across the abstraction; update create_fact mocks for (memory_data, fact_id).
Tests: 273 passed (memory suite); ruff check + format clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): address PR review - storage_path, max_facts, tracing, parsing
Six review findings (willem-bd), each verified against upstream:
- storage_path semantics (file -> root dir): migration drops file-style
(.json) legacy values with a warning; factory raises if storage_path
resolves to an existing file (avoid silent NotADirectoryError write
failure). CHANGELOG + config.example.yaml comment updated.
- create_memory_fact enforces max_facts again (via _trim_facts_to_max) and
returns (memory, None) when the cap evicts the new fact; memory_add tool
reports "not stored", client raises ValueError, POST /memory/facts -> 409.
- max_facts trim uses _coerce_source_confidence (was raw f.get("confidence",
0) -> TypeError on non-float imported/legacy confidence, swallowed as
silent update failure).
- memory-tracing assistant_id restored to "memory_agent" (was "lead-agent"
copy-paste; matches upstream + DeerMem run_name).
- _is_human_clarification_response cross-checked against
read_human_input_response (drift guard test).
- empty-string legacy values skipped silently in migration (narrow fix, not
broad "if not value" which would skip explicit bool False).
8 new regression tests. make lint + 406 memory tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): address internal review - storage fail-fast, build_llm degrade, config warn, noop template
Addresses 4 findings from the PR #4122 internal supplemental review
(parallel to willem-bd's review, no overlap):
- create_storage fail-fast: a misspelled/unimportable storage_class now
raises ValueError instead of silently falling back to FileMemoryStorage.
Memory is persistent state, so a wrong store is a data-integrity footgun;
mirrors the existing manager_class resolution policy. (storage.py)
- noop template create_fact signature: the commented template used
keyword-only `content` and returned a bare dict, while DeerMem's actual
create_fact takes positional `content` and returns tuple[dict, str|None]
(the memory_add tool passes content positionally; gateway/client/tools all
tuple-unpack). A backend copied from the template would 500 on fact-CRUD.
Template fixed; delete_fact/update_fact templates left (callers compatible).
(noop_manager.py)
- build_llm graceful degrade: wrap init_chat_model in try/except, degrade to
None + WARNING on failure (mirroring _host_default_llm) so a misconfigured
explicit model does not crash app startup -- non-LLM memory ops still work
and an update raises at runtime with the error logged. (llm.py)
- from_backend_config unknown-key warning: log a WARNING for unknown
backend_config keys (mirrors the host layer's load_memory_config_from_dict)
so a typo like `storage_pat` does not silently fall back to the default and
write memory to an unintended location. (config.py)
Tests: rewrote 3 create_storage fallback tests to expect ValueError; added 4
tests (build_llm zero-config/degrade, from_backend_config warn/silent).
make lint green; full memory suite passes.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: lllyfff <2281215061@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: lllyfff <122260771+lllyfff@users.noreply.github.com>
* fix subagent total delegation cap
* fix embedded subagent run cap context
* fix subagent cap config consistency
* fix resumed subagent run cap boundary
* fix legacy resume subagent boundary
* address subagent cap review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Update DeepSeek references from deprecated model names to the V4 lineup:
- deepseek-reasoner → deepseek-v4-pro
- deepseek-chat → deepseek-v4-flash
Keep docs and frontend mocks aligned with the wizard provider list.
* feat: emit structured runtime metadata
* fix: avoid subagent import cycle in replay gateway
* fix: preserve legacy subtask result parsing
* refactor: tighten runtime metadata contracts
* fix(middleware): keep recovery hint on task exception wrapper content
The structured-metadata stamp overwrote the wrapper text with the bare
task-failure message, dropping the model-facing 'Continue with available
context, or choose an alternative tool.' guidance that every other tool
exception keeps. Append the shared hint after the formatted message.
* fix(subagents): require lowercase hex for result_sha256 reader
Length-only validation accepted any 64-char string; a faulty serializer
or relaying wrapper could store a non-digest value in the delegation
ledger. Enforce the producer's hexdigest shape with a fullmatch.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* Add Brave image search community tool
* fix(community): length-cap Brave web_search queries
Apply _clean_query in web_search_tool so over-long queries are trimmed
to Brave's 400-char limit before the API call, matching image_search_tool
and avoiding HTTP 422 from the Brave Search API.
* fix(community): harden Brave image search SSRF guard and dimension mapping
Address PR review findings:
- Catch ValueError from urlparse so a malformed bracketed-IPv6 URL skips
one item instead of crashing the whole image_search call
- Reject IPv6 literals embedding a non-global IPv4 (IPv4-mapped, 6to4,
NAT64, IPv4-compatible), closing the loopback/private SSRF bypass
- Report width/height from the dict of the URL actually returned, so a
surviving thumbnail no longer reports the dropped original's dimensions
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Browserless web_capture screenshot tool
Add a web_capture tool that renders a page via Browserless /screenshot and
presents it through the artifact system, alongside the existing Browserless
web_fetch provider.
Hardening:
- SSRF guard: reject URLs resolving to private/loopback/link-local (incl. the
169.254.169.254 cloud-metadata endpoint)/reserved/multicast/unspecified
addresses; opt out via allow_private_addresses for internal targets.
- Surface a warning when Browserless renders a target page that itself
responded with a non-2xx/3xx status (X-Response-Code), so an error/anti-bot
page is not mistaken for valid visual evidence.
- Dedupe colliding output filenames instead of silently overwriting prior
captures.
Docs: comment out token: $BROWSERLESS_TOKEN in tool examples (an unset $VAR
fails AppConfig startup) and document allow_private_addresses.
* fix(community): format web_capture guard + document local Browserless startup
Address PR #3881 review: fix the lint-backend failure (ruff format on
browserless/tools.py) and add local Browserless startup instructions to
CONFIGURATION.md so reviewers can run the service to try web_fetch/web_capture.
Adds ``deerflow.community.e2b_sandbox.E2BSandboxProvider`` with parity
to AioSandboxProvider: metadata-keyed per-thread persistence, server-
side idle timeout, warm-pool reclaim with liveness checks, /mnt/user-data
bootstrap symlinks, dead-sandbox auto-rebuild, and release-time mirror
of agent outputs back to the host artifact directory.
Signed-off-by: joey <zchengjoey@gmail.com>
_lock was defined in __init__ and used only in reset(), leaving _apply(),
before_agent(), _clear_run_state(), and _drain_pending_warnings() unprotected.
Concurrent after_model callbacks for the same run_id could race on
usage_accum.input += diff_input, causing lost-update on token counters.
Also fix stray bare 'a' character in frontend/src/content/zh/introduction/index.mdx
that rendered as visible text on the documentation page.
Co-authored-by: kapilvus <kapilvus@gmail.com>
* feat(groundroute): add GroundRoute community web_search + web_fetch tools
GroundRoute is a meta search layer over six engines (Serper, Brave, Exa,
Tavily, Firecrawl, Perplexity) with price-based routing and failover. This
adds a self-contained community engine module (httpx only, no new required
deps) mirroring community/brave + community/tavily:
- web_search: POST /v1/search, normalize to {title,url,snippet,source_engine}.
- web_fetch: fetch a URL via mode=page.
- unit tests covering normalization, auth, clamping, and graceful errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(groundroute): register GroundRoute search + fetch in wizard and config
Add GroundRoute to the setup wizard provider lists (SEARCH_PROVIDERS +
WEB_FETCH_PROVIDERS) and as commented web_search + web_fetch examples in
config.example.yaml, mirroring tavily/serper/brave so SEARCH_API can select it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(groundroute): apply repo ruff format (line-length 240)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(groundroute): add GroundRoute to tools docs and config reference
Adds GroundRoute as a web_search and web_fetch option in the en + zh
tools.mdx pages (new tab alongside Tavily/Brave/Exa/etc.) and documents
GROUNDROUTE_API_KEY in CONFIGURATION.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(groundroute): define empty groundroute extra for clean install
The docs install line 'uv add deerflow-harness[groundroute]' (mirroring the
tavily/exa/firecrawl pattern) referenced an undefined extra, which uv accepts
but warns about. GroundRoute needs no extra packages (httpx is a core dep), so
declare an empty 'groundroute' extra in deerflow-harness optional-dependencies
so the documented command resolves without a warning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(groundroute): per-tool api key + honor caller max_results (review)
Address maintainer review on PR #3675:
- _get_api_key(tool_name): web_fetch now reads the web_fetch config block's key
instead of always web_search, so a flow that pairs GroundRoute fetch with a
different search engine authenticates correctly. Mirrors serper/exa/firecrawl.
- web_search honors a caller-supplied max_results (sentinel default None),
falling back to the configured value only when omitted, so the documented
parameter is no longer silently discarded.
- warn-once is now keyed per tool. Tests cover both fixes (web_fetch key,
agent max_results honored).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Serper Google Images provider for image_search
Add a Serper-backed `image_search` tool alongside the existing Serper
`web_search` provider, so users with a SERPER_API_KEY can pull Google
Images results as reference images for downstream image generation.
- Share request/response handling between web_search and image_search
via `_serper_post` / `_response_items`, with bounded `max_results`
(capped at 10) and query normalization.
- Add a best-effort SSRF guard (`_safe_public_url`) that rejects
non-http(s), localhost and private/non-global IP image URLs; filtered
entries are dropped and never consume the result limit.
- doctor: flag literal `api_key` values in config as a warning and steer
users toward `.env` + `$SERPER_API_KEY`.
- Docs/config: document the Serper image_search provider and SERPER_API_KEY,
and discourage committing literal keys to config.yaml.
- Tests: cover the provider end-to-end (100% line coverage on tools.py)
and the doctor literal-key warning path.
* fix(community): block obfuscated IPv4 literals in Serper image SSRF guard
The image_search SSRF guard only rejected dotted-decimal IP literals; encoded
forms such as decimal (http://2130706433/), hex (0x7f000001) and octal
(0177.0.0.1) raised ValueError in ip_address() and were allowed through, even
though many HTTP clients resolve them to private addresses like 127.0.0.1.
Add _decode_ipv4() to permissively decode these inet_aton-style encodings and
apply the same is_global check; hostnames that do not decode to an IP (e.g.
cafe.com) are still treated as hosts and left to fetch-time re-validation.
Addresses PR review feedback. Tests cover decimal/hex/octal loopback and
private encodings plus non-IP edge cases; tools.py stays at 100% line coverage.
* test(community): cover IPv4-mapped IPv6 URL filtering
* fix(community): address Serper image search review feedback
- Block trailing-dot hostname SSRF bypass (localhost./127.0.0.1.) in
_safe_public_url by stripping the FQDN root label before checks.
- Keep a filtered image/thumbnail URL empty instead of collapsing onto
its counterpart, preserving the high-res/preview contract.
- Evaluate the SSRF guard once per field rather than twice.
- Treat a null-typed organic/images field as "no results" rather than a
malformed payload.
- doctor.py: when a config $VAR is unset, fall through to the default env
var before reporting it as not set.
* feat(community): add Brave Search web search tool
Add a community web_search provider backed by the official Brave Search
API (https://api.search.brave.com/res/v1/web/search). API key is read
from the tool config (inline api_key) or the BRAVE_SEARCH_API_KEY env
var. Output schema (title/url/content) matches existing search tools.
No new dependencies (uses the existing httpx). Also wires up the setup
wizard, doctor health check, config example, and EN/ZH docs.
* refactor(community): drop redundant [:count] slice in Brave search
The Brave API already caps results via the `count` request param, so
client-side slicing was redundant. Tests now simulate the API honoring
`count` instead of relying on the slice. Addresses PR review nit.
* style(tests): apply ruff format to test_doctor.py
Collapse multiline write_text calls onto single lines to satisfy the
CI ruff formatter (lint-backend was failing on format --check).
`client.py` imported the private `_build_middlewares` from `agent.py` across a
module boundary and called it as public API. Because the `_` name signals
"module-private, no external callers", any future rename or signature change
silently breaks the embedded `DeerFlowClient` path — and the test suite even
monkeypatched `deerflow.client._build_middlewares`, baking the leak in.
`DeerFlowClient` is a lead-agent variant that genuinely needs the lead agent's
full middleware composition, so make the dependency honest: promote the helper
to a documented public entry point `build_middlewares` and update every in-repo
caller. Found during #3341 review; #3341 already removed one such leak
(`_assemble_deferred` -> public `assemble_deferred_tools`) and left this one out
of scope on purpose.
- agent.py: rename def + both internal call sites; expand the docstring into a
public-entry-point contract and document the previously-undocumented
model_name / app_config / deferred_setup params
- client.py: import + call site now use the public name (removes the last
cross-module private import)
- scripts/tool-error-degradation-detection.sh: update its import + call site
- tests (5 files): update monkeypatch/patch targets and direct calls
- docs (backend/CLAUDE.md, plan_mode_usage.md, middlewares.mdx): sync the live
references that describe the symbol as current API
Pure mechanical rename, no behavior change. Historical design docs (rfc,
superpowers spec) intentionally keep the old name as point-in-time records.
Closes#3431
* fix(loop-detection): defer warn injection to wrap_model_call
The warn branch in LoopDetectionMiddleware injected a HumanMessage
into state from after_model. The tools node had not yet produced
ToolMessage responses to the previous AIMessage(tool_calls=...), so
the new HumanMessage landed *between* the assistant's tool_calls and
their responses. OpenAI/Moonshot reject the next request with
"tool_call_ids did not have response messages" because their
validators require tool_calls to be followed immediately by tool
messages.
Detection now runs in after_model as before, but only enqueues the
warning into a per-thread list. Injection happens in wrap_model_call,
where every prior ToolMessage is already present in request.messages.
The warning is appended at the end as HumanMessage(name="loop_warning")
— pairing intact, AIMessage semantics untouched, no SystemMessage
issues for Anthropic.
Closes#2029, addresses #2255#2293#2304#2511.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(channels): remove loop warning display filter
* feat(loop-detection): scope pending warnings by run
* docs(loop-detection): update docs
* test(loop-detection): assert deferred warnings are queued
* fix(loop-detection): cap transient warning state
* docs: update docs
* add async awrap_model_call test coverage
* docs(loop-detection): document transient warnings
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat(agent): add update_agent tool for in-chat custom-agent self-updates (#2616)
Custom agents had no built-in way to persist updates to their own SOUL.md /
config.yaml from a normal chat — `setup_agent` was only bound during the
bootstrap flow, so when the user asked the agent to refine its description
or personality, the agent would shell out via bash/write_file and the edits
landed in a temporary sandbox/tool workspace instead of
`{base_dir}/agents/{agent_name}/`.
Changes:
- New `update_agent` builtin tool with partial-update semantics (only the
fields you pass are written) and atomic temp-file + os.replace writes so
a failed update never corrupts existing SOUL.md / config.yaml.
- Lead agent now binds `update_agent` in the non-bootstrap path whenever
`agent_name` is set in the runtime context. Default agent (no
agent_name) and bootstrap flow are unchanged.
- New `<self_update>` system-prompt section is injected for custom agents,
instructing them to use `update_agent` — and explicitly NOT bash /
write_file — to persist self-updates.
- Tests: 11 new cases in `tests/test_update_agent_tool.py` covering
validation (missing/invalid agent_name, unknown agent, no fields),
partial updates (soul-only, description-only, skills=[] vs omitted),
no-op detection, atomic-write safety, and AgentConfig round-tripping;
plus 2 new cases in `tests/test_lead_agent_prompt.py` covering the
self-update prompt section.
- Docs: updated backend/CLAUDE.md builtin tools list and tools.mdx
(en/zh) with the new tool description.
* feat(agent): isolate custom agents per user
Store custom agent definitions under the effective user, keep legacy agents readable until migration, and cover API/tool/migration behavior with tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: consistent write/delete targets & add --user-id to migration
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
- Added titles and descriptions to workspace usage, configuration, customization, design principles, installation, integration guide, lead agent, MCP integration, memory system, middleware, quick start, sandbox, skills, subagents, and tools documentation.
- Removed outdated API/Gateway reference and concepts glossary pages.
- Updated configuration reference to reflect current structure and removed unnecessary sections.
- Introduced new model provider documentation for Ark and updated the index page for model providers.
- Enhanced tutorials with titles and descriptions for better clarity and navigation.
* feat(blog): implement blog structure with post listing and tagging functionality
* feat(blog): enhance blog layout and post metadata display with new components
* fix(blog): address PR #1962 review feedback and fix lint issues (#14)
* fix: format
---------
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
* feat: add docs site
- Implemented dynamic routing for MDX documentation pages with language support.
- Created layout components for documentation with a header and footer.
- Added metadata for various documentation sections in English and Chinese.
- Developed initial content for the DeerFlow App and Harness documentation.
- Introduced i18n hooks and translations for English and Chinese languages.
- Enhanced header component to include navigation links for documentation and blog.
- Established a structure for tutorials and reference materials.
- Created a new translations file to manage locale-specific strings.
* feat: enhance documentation structure and content for application and harness sections
* feat: update .gitignore to include .playwright-mcp and remove obsolete Playwright YAML file
* fix(docs): correct punctuation and formatting in documentation files
* feat(docs): remove outdated index.mdx file from documentation
* fix(docs): update documentation links and improve Chinese description in index.mdx
* fix(docs): update title in Chinese for meta information in _meta.ts
