91 Commits

Author SHA1 Message Date
Totoro
b0bca9dcb4
feat(mcp): bind HTTP tool credentials to approved custom agents (#8242)
* feat(mcp): bind agent credentials to approved definitions

* fix(mcp): enforce live auth policy and pin task destinations

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-11-14 22:10:33 +08:00
hataa
7892f79f9c
fix(mcp): drop cross-server duplicate bound tool names instead of silently shadowing (#8135)
Server-prefixed tool names are not unique across servers: server 'a'
exposing 'b_x' and server 'a_b' exposing 'x' both bind as 'a_b_x'. The
flatten step forwarded both declarations to the model, where the provider
silently keeps only one and the other server's tool becomes unreachable
with no diagnostic.

Deduplicate by bound name in enabled-server declaration order and drop
later duplicates with a warning.

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-12 19:07:32 +08:00
Wenchao An
3d826c6d95
feat(approvals): add Ask, Auto and Full tool permission modes (#8033)
* feat(hitl): add human-in-the-loop approval for high-risk tool calls

* fix(hitl): address approval review findings

* docs(hitl): keep agent guidance under size limit

* fix(hitl): guard approval pause recovery

* docs(hitl): shrink inherited guidance

* fix(hitl): harden approval admission and recovery

* fix(approvals): address review safety and reconciliation gaps

* ci: retry approval checks after runner timeout

---------

Co-authored-by: wangzeren <1004695029@qq.com>
2026-11-11 20:59:05 +08:00
Xinmin Zeng
2a97ec3e58
docs: turn README into an index and move reference detail to docs/guide (#8113)
* docs: turn README into an index and move reference detail to docs/guide

The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md.

* docs: send user docs to topic guides and budget the root README

The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003).

* chore: remove unreferenced screenshots and finished plan notes

pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them.

* docs: route always-loaded AGENTS.md files to scoped guides

* docs(agents): cut scoped guides to scope, verify, traps, decisions

Each AGENTS.md keeps four kinds of content: the scope of the directory, the
verification commands, traps a local test run does not show, and decisions the
code cannot state. Values, field lists, function inventories, test names, issue
numbers and history are deleted, because the code, the tests and git state them.

Search and fetch providers share one community guide. Four guides with no
remaining content are removed. The size check now allows 6 KiB per file (hard
limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB).

* style: format frontend guide table and guidance test imports

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-11 13:26:17 +08:00
hdu_zhou
2feec9cb09
fix(mcp): degrade a blob without mimeType instead of failing the tool call (#7800)
* fix(mcp): degrade a blob without mimeType instead of failing the tool call

A resource link or embedded resource without mimeType made the blob guard
raise, so the whole tool call failed instead of degrading the attachment.
Keep the guard for genuinely unusable payloads and let a missing media type
fall through to the caller's default.

* Modify test for text blocks and mime type validation

Update test to check for text blocks and their mime types.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* test(mcp): remove MIME assertion from text content block

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-11-05 21:40:32 +08:00
asts
59beff5799
fix(mcp): load personal transport TLS trust off the event loop (#7909) 2026-11-05 14:27:52 +08:00
alexcastgonz
31e7b4ae4f
fix(mcp): keep a tool argument named runtime out of LangGraph's runtime injection (#7677)
* fix(mcp): keep a tool argument named runtime out of LangGraph's runtime injection

The pooled stdio wrapper and the background-submit wrapper took LangGraph's
ToolRuntime in a parameter called `runtime`. ToolNode injects by that name, so a
tool whose own input schema has a `runtime` property had the model's value
replaced by the ToolRuntime and then lost: the server received the other
arguments only, and TaskSubmitRequest.arguments missed it too.

Name the injected parameter `deerflow_runtime` in both wrappers (it is still
detected through its ToolRuntime annotation) and read it under that name in the
personal-MCP ownership guard. Existing tests that call the wrappers pass the new
name; two ToolNode tests cover a `runtime` business argument on each path.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(mcp): read the injected runtime under the right name in the personal-tool guard

The personal-tool ownership guard read kwargs["deerflow_runtime"], but only
the pooled stdio wrapper was renamed; personal HTTP/SSE tools still carry
langchain-mcp-adapters' `runtime` parameter, so the guard never saw the
authoritative runtime user for them. Look the parameter name up per tool, and
update the mcp AGENTS.md note on how the runtime is injected.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 12:45:14 +08:00
NanPan
5c4ba4d2fe
fix(mcp): drain compensation work during service shutdown (#7662)
* fix(mcp): drain compensation work during service shutdown

* test(mcp): make shutdown drain assertion deterministic

* fix(mcp): preserve shutdown drain through caller cancellation

* style(mcp): keep test module spacing consistent

* fix(mcp): report pending compensation tasks on shutdown timeout

* test(mcp): cover compensation observability on poller timeout
2026-10-29 14:40:09 +08:00
asts
f79debed59
fix(mcp): reject unsupported inline audio results (#7628)
* fix(mcp): reject unsupported inline audio results

* fix(mcp): include MIME type in unsupported audio errors
2026-10-28 14:00:34 +08:00
yi-coding
c04946a73f
fix(mcp): retain durable control claims through slow calls and cancellation (#6780)
* fix(mcp): retain durable control claims through slow calls and cancellation

* docs(mcp): distinguish poll and cancel claim fencing

---------

Co-authored-by: xyi74976-del <290725798+xyi74976-del@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-21 22:03:21 +08:00
Nan Gao
f39cdd6077
fix(mcp): keep rotating OAuth refresh tokens across restarts and Gateway instances (#7207)
* fix(mcp): keep rotating OAuth refresh tokens across restarts and Gateway instances

MCP servers on the OAuth refresh_token grant lost their rotated refresh
token whenever a new process or manager started (restart, tool-cache
reset for non-task servers, every personal-connection run, a second
Gateway instance) and re-seeded with the consumed configured token,
getting invalid_grant for good.

- New table mcp_oauth_tokens (migration 0047_mcp_oauth_tokens), one row
  per refresh_token connection keyed (scope, owner_user_id, server_key):
  ('deployment', '', name) or ('personal', owner, personal runtime
  name). Access and refresh tokens are DEER_FLOW_CREDENTIALS_KEY
  ciphertext; a config fingerprint (token_url, client_id, grant, scope,
  audience, seed) makes a changed seed win.
- OAuthTokenManager re-reads the row under its process lock, adopts a
  peer's valid token, and otherwise claims the refresh with a lease plus
  a version compare-and-set (database clock), fetches outside any
  transaction, writes under the lease and releases it on failure.
- The Gateway installs the store (sync engine via to_thread) around the
  LangGraph runtime for sqlite/postgres; personal managers pass their
  owner. client_credentials stays in process; an unloadable key falls
  back to process memory; database errors fail the refresh.
- credentials_key_consumers lists refresh_token MCP OAuth servers, so a
  declared multi-instance deployment needs the shared key.

Docs: MCP_SERVER.md, CONFIGURATION.md, Helm README, README, mcp and
migrations AGENTS.md, CHANGELOG (en/zh).

* docs(changelog): link the MCP OAuth token store entry to #7207

* fix(mcp): prune removed connections' OAuth token rows and ask owners to re-authorize on invalid_grant

Review follow-up (willem-bd):
- Both MCP config writers drop mcp_oauth_tokens rows of connections a
  save removes (deployment: _commit_mcp_config_write; personal:
  _mutate, keeping each connection's enabled/disabled runtime-name
  variants so a toggle keeps its chain). Best effort; no startup sweep,
  since a missing config file would look like every server removed.
- invalid_grant on a refresh_token request raises
  McpOAuthReauthorizationRequired: the tool error names the server and
  asks for a new refresh token, plus the shared-key hint when this
  instance could not decrypt the stored chain.
- Annotate the gateway helpers' config as AppConfig.

* fix(mcp): renew the OAuth refresh lease, keep unstored rotations, and key access tokens by request parameters

Review follow-up (willem-bd, second round):
- HTTPX timeouts bound each I/O wait, not a whole response: the refresh
  holder now renews its lease every third of REFRESH_LEASE_SECONDS while
  the token request runs and abandons it after REFRESH_MAX_SECONDS (120 s);
  waiters outlast a live holder. Regression uses a real loopback endpoint
  that streams its body for three lease lengths.
- A refresh whose tokens could not be stored keeps them pending
  (unsaved_base_version): they are written back before an older stored
  token is adopted or refreshed with, retried every few seconds while the
  access token is still valid, and dropped once a peer stored a newer
  chain. A failed save releases the lease.
- New token_request_fingerprint (extra_token_params and response fields):
  a stored access token is reused only under the same request parameters;
  otherwise the stored refresh-token chain is refreshed.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-21 18:17:29 +08:00
ordinary being
eed7fa8f05
fix(mcp): fail durable tasks whose connection no longer resolves (#7058)
* fix(mcp): fail durable tasks whose connection no longer resolves

A personal MCP runtime name is derived from the server's complete definition, and a deployment task server belongs to the Gateway startup snapshot. Editing, disabling, or deleting a personal server while a durable task is in flight therefore makes the persisted server name unresolvable. McpTaskToolCaller raised LookupError, which the service classifies as transient: status polls and cancellation retried forever, leaving the task stuck in working and impossible to cancel.

Raise a dedicated McpTaskConnectionUnavailableError instead, and terminalize it: status polls persist failed with the connection error and stop; a pending cancellation persists failed locally because the remote task can no longer be reached. Transient status/cancel errors keep their existing capped-backoff retry policy.

Tests: new service tests pin both terminal paths, and caller tests pin the exception for missing deployment and stale personal server names.

* fix(mcp): preserve LookupError compatibility for connection errors

McpTaskConnectionUnavailableError now subclasses LookupError instead of RuntimeError. The four existing personal-MCP assertions in test_personal_mcp.py still treat a missing or stale connection as a lookup failure; keeping that base preserves their behavior and avoids breaking callers that caught the previous type, while the task service still catches this exact type for permanent-failure classification.

Add a compatibility test and rerun the personal-MCP and task suites.

---------

Co-authored-by: Tong-bit-art <Tong-bit-art@users.noreply.github.com>
2026-10-19 11:32:28 +08:00
asts
64dab5897a
fix(mcp): preserve remote URLs during local file rewriting (#6878)
* fix(mcp): preserve remote URLs during local file rewriting

* fix(mcp): honor escaped delimiters in remote URLs
2026-10-17 08:13:35 +08:00
Hyeonsang Cho
e02d5e2fc3
fix(mcp): skip tasks a cancel released while the poll claim was in flight (#6859)
* fix(mcp): skip tasks a cancel released while the poll claim was in flight

claim_due_tasks selected due rows with FOR UPDATE SKIP LOCKED, then set
the lease through an ORM flush that updates by id alone. FOR UPDATE is a
no-op on SQLite, so a request_cancel that committed between the SELECT
and the flush had its cleared poll lease written back. The cancellation
worker only claims rows without a live lease, so the remote cancel waited
for mcp_tasks.lease_seconds (120 s by default), and the task was polled
once more for a result apply_snapshot then discarded.

The claim now selects candidate ids and leases each one with a
conditional UPDATE that re-checks the claim predicates (pollable status,
no cancel request, due next_poll_at, no live lease), returning only the
rows it actually leased, the same atomic fence release_claim already
uses. On Postgres the row lock still orders the two writers.

* docs(changelog): reference #6859 for the MCP task claim fence

* docs(mcp): note that a short poll claim batch is not topped up
2026-10-15 09:42:46 +08:00
RongJie G
44a896cc12
fix(mcp): preserve active calls during session eviction (#6900)
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
2026-10-14 22:55:43 +08:00
spud
eadc1e230e
feat(mcp): reuse unchanged per-server tool discovery results (#6795)
* feat(mcp): add per-server discovery cache scaffolding

Introduce the per-server tool cache entry, canonical parsed-config snapshots on the applied revision, and the shared _binding_matches reuse predicate. Identity reuse is object-identity based on the pool's current binding (no value equality); snapshots and ownership handles stay out of repr. No selection, eviction or publication is wired yet.

* feat(mcp): selectively evict stale per-server tool cache entries

Evict per-server discovery results during selective reconciliation, after the new binding epoch is installed, so unchanged servers keep their exact entry objects while changed, removed or re-bound servers (including lifecycle-token rotations) are dropped. Delegate the binding verdict to _binding_matches so reuse and eviction cannot diverge.

Clear _server_tool_cache on every full reset, and count retained entries in the reconciliation state predicates so a lingering entry can never let a later config change be silently skipped.

* feat(mcp): reuse unchanged per-server results during initialization

Initialize MCP tools from per-server discovery instead of a single full pass: select reusable entries under the claim lock, discover only the missing servers with get_mcp_tools_by_server, then publish atomically after re-validating every candidate binding against the pool's current binding via _binding_matches. Failed servers stay absent (never cached as empty), and superseded or partially bound candidates are discarded whole. Test fixtures now drive the grouped entry point.

* fix(mcp): close T4 review gaps in per-server cache publication

Address four findings from the adversarial review: treat retained entries as unverifiable when the applied baseline is None so an interceptor change still forces a full reset; count retained entries in the claim-time shared-reset predicate; refuse to publish a false-success empty cache when every enabled server failed discovery; and hoist the lazy imports inside the initializer's try/finally so an import failure releases the initialization claim.

* fix(mcp): sanitize cache failure logging and document per-server cache

The outer lazy-initialization handler now logs only the exception type, never the raw message or traceback, so a discovery error cannot print a resolved $VAR credential. The committed-writer planner mirrors the reader's no-baseline guard so retained entries are never reused under an unknown interceptor chain. Document the per-server tool cache (reuse, invalidation, failure handling, remote tool changes) in MCP_SERVER.md and mcp/AGENTS.md, and drop phase-staging wording from the new code and tests.

* docs(mcp): keep the agent-guidance chain under its hard limit

F2's cache notes pushed the effective AGENTS.md instruction chain 905 bytes past its 98304-byte hard limit, failing the agent-guidance CI job. Compress the selective-reconciliation bullet to its invariants (the full cache behaviour lives in docs/MCP_SERVER.md), and reword the total-failure comment so it no longer implies that build-invalid connections were attempted.

* fix(mcp): fence cached tools by interceptor chain

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-14 08:44:21 +08:00
spud
9fd88c5ffc
fix(mcp): avoid leaking credentials in discovery error logs (#6794)
* fix(mcp): keep resolved credentials out of discovery failure logs

A failed per-server MCP discovery logged the raw exception message and traceback. ExtensionsConfig resolves $VAR values before validation, so a discovery error can embed a resolved-credential request URL (for example an httpx error carrying a URL with userinfo). Log only the server name and the exception type instead, at both the per-server and whole-phase handlers, and pin it with a caplog regression test using a simulated credential.

* test(mcp): assert the exact retained server name when redacting logs

The redaction regression test asserted a bare "A" in the captured log, which any unrelated A would satisfy. Assert the full "server 'A'" fragment so the test actually proves the offending server name is preserved while the credential is not.

* test(mcp): assert structured discovery warning arguments

The redaction change made the per-server discovery warning a parameterized log call, so asserting the server name inside the raw format string no longer matches. Assert the structured arguments instead: the server name, the exception type, the fully formatted message, and the absence of exc_info.
2026-10-14 06:40:20 +08:00
spud
3343522fa2
refactor(mcp): expose per-server tool discovery results (#6734)
* refactor(mcp): expose per-server tool discovery results

* refactor(mcp): clarify per-server discovery contracts
2026-10-13 08:38:36 +08:00
spud
75ce6f4920
fix(mcp): preserve server binding history across gateway workers (#6617)
* fix(mcp): preserve server binding history across gateway workers

A peer Gateway worker could keep a stale pooled session after another worker deleted and identically re-added a stdio server, because the final effective MCP slice matched the revision it had already applied.

Record a per-server lifecycle generation in the shared extensions config and consume it in the reader, discovery claim, and committed handoff so an observed generation change force-rebinds only the affected server. Legacy configs derive a stable in-memory token; deletions leave permanent tombstones.

* fix(mcp): fail closed on an invalid mcpLifecycle instead of retrying

An unverifiable mcpLifecycle made initialize_mcp_tools() unpublishable, so get_cached_mcp_tools() retried discovery forever and could keep respawning stdio servers. Abort discovery on an invalid ledger after conservatively retiring local MCP state; a fixed config recovers on the next call.

Also documents the rolling-upgrade prerequisite: cross-worker ABA detection requires configuration writes to go through a lifecycle-aware writer.

* refactor(mcp): share stdio fingerprinting across lifecycle paths

Reuse lifecycle._stdio_fingerprints from the applied-revision builder so the connection-identity rule has a single implementation, and move the unusable-server diagnostic into the shared helper.

Add a regression proving the Gateway Skills writer preserves mcpLifecycle byte-for-value, and document that DeerFlowClient.update_mcp_config can raise RuntimeError after the config was committed.

* docs(mcp): condense lifecycle guidance under the AGENTS chain limit

The effective mcp/AGENTS.md chain was 15 bytes under the 98,304-byte CI hard limit, so the standalone cross-worker lifecycle bullet pushed it over. Fold the mcpLifecycle and rolling-upgrade facts into the existing selective-reconciliation bullet and condense it, keeping the chain under the limit while preserving the contract.

* docs: trim inherited agent guidance below CI size limit

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-11 11:45:28 +08:00
Storm
588ad010c0
fix(mcp): stop persisting URL-sourced ResourceLink blocks and heal bricked threads (#6606)
* fix(agents): sanitize persisted URL content blocks before model calls

Threads whose checkpoints carry URL-sourced file blocks (written before
the MCP conversion-layer fix) fail every model call at serialization: the
Chat Completions translator rejects any file block carrying a url, and the
error handler answers each later turn with a non-retriable fallback. Add a
read-time sanitizer that rewrites those blocks in the model-bound request
view only, so poisoned threads heal on their next turn with no migration
and no user action.

- ModelContentCompatibilityMiddleware wraps the model call (sync and
  async) and rewrites ToolMessage list content: URL-sourced file blocks
  become "[Resource ({mime or unknown type}) available at {url}]" text
  blocks, and image blocks whose scheme is not http/https/data get the
  same placeholder. Fetchable image URLs, base64 payloads, text, and
  non-ToolMessage messages pass through. The rewrite is order-preserving,
  idempotent, and never writes state or mutates checkpoints, so artifact
  capture and other state readers keep seeing the original blocks.
- Register on all three chains that replay checkpointed ToolMessages:
  build_middlewares (lead; outer of the MODEL_PHYSICAL extension anchor so
  its final-request guarantee holds), the SDK factory
  _assemble_from_features (new step 13a; docstring updated and its stale
  omission of TokenBudgetMiddleware fixed), and
  build_subagent_runtime_middlewares (after DurableContext/summarization,
  before the date/coalescer tail).
- Subagent coverage verified by tracing every subagent agent-creation
  site: executor.py _create_agent is the only caller of
  build_subagent_runtime_middlewares and serves both the task tool and the
  durable batch worker (SubagentRuntime itself only carries dependencies),
  and a wiring test now asserts the middleware is present in that
  builder's output; the lead and factory chains gained the same wiring
  assertions.
- Update the chain-order contract tests for the new slot: full-chain
  order, loop-detection adjacency, custom loop-detection tail positions,
  and the subagent chain count (22 -> 23).

* fix(mcp): stop emitting URL-sourced file blocks from ResourceLink results

MCP ResourceLink conversion produced a URL-sourced file content block for
every non-image link, which LangChain's Chat Completions translator rejects
at serialization time. Once such a block was checkpointed in the
ToolMessage, every subsequent turn of the thread failed at model-call time,
permanently bricking the conversation.

- Downgrade every ResourceLink that is not an http(s) image to a short text
  placeholder carrying the resource name, MIME type, and the
  already-virtualized URL, so host paths never leak and the model still
  learns a resource was produced and where it lives.
- Preserve each downgraded link as structured data in the tool-message
  artifact under resource_links (merged with structured_content when
  present) for artifact capture and future MCP Apps rendering.
- Keep http(s) image links as image blocks; embedded base64 payloads, text
  path rewriting, and the error-to-ToolException path are unchanged.
- Re-contract the file-migration tests to the new behavior and document
  the conversion contract in the MCP AGENTS.md.

* docs(middlewares): add ModelContentCompatibilityMiddleware to the chain inventory

Insert the read-time sanitizer at its actual lead-chain position (after the
token-budget guard, before the custom-middleware injection point) and renumber
the tail entries 34-39 to 35-40.

* fix(agents): rewrite file blocks on url key presence, not truthiness

langchain-core's OpenAI translator raises for ANY file block carrying a
url key — an empty string or None included — so the read-time sanitizer
must not let such blocks through just because the value is falsy. A
persisted block with url="" or url=None would still brick every later
model call in the thread.

- Gate the file-block rewrite on "url" in block instead of a truthiness
  check; a non-empty string URL keeps the
  "[Resource (mime) available at url]" placeholder, while an empty or
  None URL gets a location-less "[Resource (mime)]" placeholder, so no
  unusable location is advertised to the model.
- Pin both shapes with new middleware tests (url="" and url=None file
  blocks are both rewritten to text). The image branch is unchanged: an
  empty-image case fails provider-side with a 400, not a bricked thread.

* fix(mcp): keep data URIs out of model text and resource_links

A data: scheme ResourceLink fell into the generic placeholder branch,
inlining megabytes of base64 into the model-visible text and into
resource_links, which is checkpointed state — against the repo policy
that data:/blob: URIs never enter state.

- Special-case data: links in _convert_call_tool_result: the
  model-visible placeholder becomes "[Resource: name (mime) embedded
  inline]" and the raw URI is not added to resource_links at all, so the
  payload can neither bloat the context window nor persist in the
  checkpoint.
- Pin the behavior with a conversion test asserting both the placeholder
  wording and the absence of the payload/URI from every emitted block and
  the artifact.
- Add the missing passthrough pin for an HTTPS:// uppercase-scheme image
  link: scheme matching is case-insensitive, so the link stays an image
  block (the block's URL validation normalizes the scheme to lowercase).

* fix(tools): consume resource_links in extract_artifacts_from_result

The MCP conversion layer preserves downgraded ResourceLinks as
{"name", "uri", "mime_type"} entries under the tool message
artifact's resource_links key, but the artifact extractor only read
structured_content — a result whose only artifact signal was
resource_links produced no entries and the file silently vanished from
the artifacts panel.

- Add a fourth extraction path: each resource_links entry becomes a file
  artifact entry, reusing _is_referenceable_url like the
  structured-content and content-block paths, so data:/blob: and other
  unresolvable URIs stay rejected per existing policy. The entry is named
  after the link when a name is present, falling back to the URI's final
  segment; a string mime_type is carried through.
- Integration-test a result whose only signal is resource_links, plus the
  referenceability gate (data:, ui://, and absolute-path URIs in one
  artifact).

* docs(mcp): update the ResourceLink conversion contract bullet

Mention the new artifact-registry consumer and the data: special case so
the bullet stays accurate: uppercase http(s) schemes match
case-insensitively, data: URIs get the embedded-inline placeholder and
are dropped from resource_links, and extract_artifacts_from_result turns
resource_links into registry entries through the shared
referenceability gate.

* refactor(tools): add the canonical resource placeholder formatter

Both the MCP conversion layer and the read-time
ModelContentCompatibilityMiddleware downgrade model-incompatible resource
references to a text placeholder, but each built its own literal. Extract
the single formatter they will share so a thread healed at read time shows
the model the same placeholder shape a fresh conversion produces; the name
segment is optional because persisted content blocks carry no resource name.

* fix(mcp): keep host paths out of ResourceLink placeholder text

Review follow-up on the ResourceLink conversion contract. The placeholder
embedded the resolved URL unconditionally, so a link outside the user-data
tree (or converted without thread context) leaked a raw host path — a
file:// URI or bare filesystem path — into model-visible text, violating
US-16. Compute the resolved URL's scheme and withhold the location segment
when it spells out a host filesystem location (empty scheme, file, or a
single-letter Windows drive scheme); virtual paths and remote schemes keep
their location. The unresolved URI still lands in the artifact channel,
which the model never sees.

- add _model_visible_link_location gate and route the generic placeholder
  through it, re-contracting the outside-user-data and no-context tests to
  the location-less shape and pinning the Windows-drive case
- treat blob: like data: as non-referenceable inline input: drop its raw
  URI from checkpointed resource_links and emit the location-less
  placeholder instead of the mismatched 'embedded inline' wording, and fix
  the comment that claimed blob: was already excluded
- replace every inline placeholder f-string in the ResourceLink branch
  with the shared resource_placeholder_text formatter so the conversion
  layer and the read-time middleware emit byte-identical shapes; data:
  links now render in the canonical location-less form
- sync the ResourceLink conversion contract bullet in mcp/AGENTS.md and
  the _convert_call_tool_result docstring with the host-path suppression
  rule and blob: handling

* fix(agents): sanitize URL content blocks on any message role

Address the second-pass review of the request sanitizer with two
changes that share the same code path.

- Adopt the canonical resource_placeholder_text formatter from
  deerflow.tools.resource_placeholder instead of the module-local
  placeholder helpers, so the read-time rewrite and the MCP conversion
  layer emit byte-identical placeholder text from one definition.
- Widen the rewrite from ToolMessage-only to every request message with
  list-form content. User uploads are verified to produce base64 blocks,
  but imported or cross-client history can carry a URL-sourced file
  block on a HumanMessage or AIMessage, and the Chat Completions
  serializer rejects such a block regardless of the role carrying it.
  The rewrite stays request-view-only: state and checkpoints are never
  mutated, and base64/file-id blocks and string content pass through
  untouched on any role.

* docs(agents): state the ModelContentCompatibility placement contract

The comment above the registration claimed the sanitizer sits just
before Clarification. In the lead chain it is actually followed by the
custom/extension injection point and the whole terminal tail
(terminal-response, finish-reason, and the always-last
ClarificationMiddleware). Describe the real contract: after every
built-in content transform, before extension injection and the
terminal tail.

* docs(middlewares): clarify provenance stamps for request-view-only rewrites

The message-provenance rule read as if every rewrite needs a
provenance_kwargs stamp. Add one line noting the exception: rewrites
that only modify the model-bound request view and never enter state
need no stamps, citing InputSanitizationMiddleware,
DanglingToolCallMiddleware, and ModelContentCompatibilityMiddleware as
precedents.

* fix(agents): keep host paths and data URIs out of healed placeholders

The read-time ModelContentCompatibilityMiddleware rewrote URL-sourced
file blocks and non-fetchable image blocks into placeholders that
inlined the block's URL verbatim. Pre-fix checkpoints — exactly the
threads this middleware exists to heal — can carry a file:// URI, a bare
or Windows host path (links that never resolved to virtual paths), or a
data: URI embedding megabytes of base64, so healing leaked deployment
filesystem layout into prompts (US-16) and could inject the whole inline
payload into model-visible text.

The conversion layer already suppresses those forms via
_model_visible_link_location. Move that gate into the shared
deerflow.tools.resource_placeholder module as model_visible_location
(now also collapsing data:/blob: URIs) and route both middleware
branches through it, so both layers apply identical suppression rules
with one definition. Conversion behavior is unchanged: its data:/blob:
branch short-circuits before the gate, and the gate covers those schemes
for defense in depth.

- resource_placeholder.py gains model_visible_location alongside the
  shared formatter; mcp/tools.py drops its private copy and imports it.
- The middleware's file and image branches pass the URL through the gate
  before building the placeholder; module docstring documents the rules.
- Tests: gate unit tests (virtual/remote visible; file://, bare/Windows
  host paths, data:/blob: withheld) and middleware regression tests
  pinning that a pre-fix data: payload, file:// URI, and Windows drive
  path heal to location-less placeholders.
- AGENTS.md bullets in mcp/ and middlewares/ state the shared gate.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(agents): downgrade malformed image URLs instead of raising in the sanitizer

The read-time ModelContentCompatibilityMiddleware gated persisted image
blocks on urlparse(url).scheme, but urlparse raises ValueError on
structurally invalid URLs — a bad IPv6 literal, invalid netloc
characters — instead of returning an unusable scheme. Pre-fix
checkpoints and imported history can carry exactly those URLs, and the
block stays in the checkpoint, so every later model call in the thread
raised again: the sanitizer re-bricked the very threads it exists to
heal, only with a framework-side exception instead of the langchain
serialization error.

Catch ValueError and treat the URL as non-fetchable, so the block
degrades into the same text placeholder as any other unresolvable
scheme (the shared model_visible_location gate already swallows
ValueError, so the location segment is withheld as well).

- wrap the image branch's urlparse call in try/except ValueError and
  fall back to an empty scheme, keeping http/https/data pass-through
  behavior unchanged for well-formed URLs
- add a regression test pinning that an image block with a bad IPv6
  literal heals to the location-less placeholder without raising

Co-Authored-By: Codex <noreply@openai.com>

* fix(tools): dedupe artifact refs across extraction sources

One downgraded ResourceLink produced two registry entries: the typed
resource_links channel added an entry for the URI, and the free-text
scan over the conversion layer's placeholder text — which embeds the
same virtual path or remote URL — matched it again. The two handles
pointed at one resource, spent double of the render budget, and showed
the model two handles for a single artifact.

extract_artifacts_from_result now tracks seen real_refs across all
sources and keeps only the first entry per ref. Extraction order makes
the typed sources (structured content, resource_links, content blocks)
win over the free-text scan, so the surviving entry keeps the link's
display name and mime type.

- route all six entry-producing sites through a local _add_unique
  helper keyed on real_ref; per-source behavior is unchanged
- document the cross-source dedup in the function docstring
- add a regression test covering a virtual-path and a remote-URL
  placeholder together with their resource_links entries

Co-Authored-By: Codex <noreply@openai.com>

* docs(mcp): state the intentional data: image downgrade asymmetry

The conversion layer downgrades image/* data: ResourceLinks to
location-less text placeholders, while the read-time middleware lets
persisted data: image blocks pass through to the model. The two
contracts looked contradictory but are deliberately layered: the
conversion layer keeps new inline payloads out of checkpointed state,
and the middleware only preserves blocks already occupying one.

- mcp/AGENTS.md's ResourceLink conversion contract and the conversion
  comment in mcp/tools.py now state the image/* data: downgrade and
  why it differs from the read-time pass-through
- middlewares/AGENTS.md's ModelContentCompatibilityMiddleware entry
  states the symmetric read-time rule
- pin the behavior with an image-mime data: ResourceLink conversion
  test (location-less placeholder, nothing in the artifact channel)

Co-Authored-By: Codex <noreply@openai.com>

* docs(backend): compact AGENTS.md guidance to fit the chain byte budget

The branch's new contract bullets pushed the middlewares and mcp
effective AGENTS.md chains past the 98304-byte AG002 hard limit (the
sandbox/subagents chains on main were already over it). Compact the
guidance without dropping contracts:

- middlewares item 34 points at the MCP ResourceLink conversion
  contract and the module docstring instead of restating both
- the mcp ResourceLink conversion contract bullet is tightened
- backend/AGENTS.md merges the File Upload and Context Summarization
  bullet sprawl, folds the stray ATX test note into the upload outline
  rule, drops the Nginx routing block the root guide already covers,
  merges the duplicate MCP-task pointer bullet, compacts Vision
  Support, and scopes the Project Structure tree to the backend
  subtree (the root guide maps the full repo)

Co-Authored-By: Codex <noreply@openai.com>

* fix(agents): gate the file-URL downgrade on provider support and neutralize placeholder metadata

Two review findings on the read-time sanitizer:

- The file-block downgrade ran for every provider, but only Chat
  Completions rejects URL-sourced file blocks: the OpenAI Responses API
  (input_file file_url) and Anthropic (document url source) accept
  http(s) document URLs, so a persisted block created for those
  providers lost its document from the model request. Keep the block
  when the selected model serializes http(s) file URLs natively; every
  other URL shape (non-http schemes, empty/None, malformed) still
  degrades to the placeholder.
- The placeholder interpolated persisted MIME/location metadata
  verbatim, but the middleware runs after the input/tool-result
  sanitization passes and historical URL blocks never went through them
  (non-text content is left untouched), so values carrying
  <system-reminder>-style tags or user-input boundary markers reached
  the model raw. Route the interpolated metadata through the shared
  neutralize_untrusted_tags guard in both the file and image branches.

Tests: provider gating (Responses/Anthropic preserve, Chat Completions
still downgrades, non-http and malformed URLs downgrade even for
capable providers) and metadata neutralization (injection tags
HTML-escaped, boundary tokens neutralized, file and image branches).

Co-Authored-By: Codex <noreply@openai.com>

* fix(tools): require a directory boundary before exposing virtual locations

model_visible_location admitted any URL starting with the virtual
prefix, so a sibling like /mnt/user-data-backups/ops/private/report.pdf
was treated as a virtual path and its host location reached
model-visible placeholder text — defeating the US-16 host-path
suppression the gate exists for. Require the exact virtual root or a
slash-delimited descendant; siblings now collapse to the location-less
placeholder like any other host path. Both the file and image
downgrade branches share the gate, and the conversion layer applies
the same rules to fresh results.

Regressions: gate unit tests (exact root and descendants stay visible;
/mnt/user-data-backups and /mnt/user-database siblings withheld) and
middleware tests pinning the location-less placeholder for both
branches.

Co-Authored-By: Codex <noreply@openai.com>

* docs(agents): scope the placeholder artifact-visibility claim to the conversion layer

The module docstring claimed the healed placeholder keeps the location
visible to the artifact free-text scan and that the link survives in
the structured artifact channel — true for the conversion layer, whose
placeholders are checkpointed, but not on the healing path: the request
view never reaches artifact capture, and no resource_links entry is
created for a downgraded block. Spell out what actually happens here:
a referenceable original block (http(s) URL or absolute host path) is
still captured, from the capture-time scan of the state block rather
than the placeholder, while a non-referenceable one (file://, data:,
blob:) yields no entry and survives only in the untouched checkpoint.

Co-Authored-By: Codex <noreply@openai.com>

* docs(mcp): clarify healing and artifact preservation contracts

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-10 21:23:27 +08:00
spud
cd15687284
fix(mcp): reconcile committed Gateway config mutations (#6567)
* fix(mcp): reconcile committed Gateway config mutations

Install selective MCP binding fences inside the Gateway config commit boundary and finish teardown after releasing config locks. Reject frozen durable-task server changes before persistence.

Preserve drained cancellation semantics and cover delete/re-add, durable-only bindings, multi-server diffs, shared writers, and post-write reconciliation failure.

* test(mcp): pin committed handoff boundaries

Prove the fallback fence is installed before releasing the config lock, a committed writer cannot swallow an earlier shared reset, delete/re-add advances the applied baseline and cache generation, and retries after a post-write 500 have explicit conflict or missing semantics.

* fix(mcp): fence committed config after reload failure

Treat a post-write reload failure as a committed handoff failure so local MCP state is detached before releasing the config lock and teardown remains outside it.

* fix(mcp): always finish committed pool teardown

Run finish_mcp_reconciliation from a post-lock try/finally so response or logging failures cannot strand a retired full-reset pool.

* test(mcp): fence cold-start binding race

Pin the in-flight discovery path that keeps the committed fast path from admitting a late old-fingerprint binding.

* docs(mcp): document committed teardown guarantee

* fix(mcp): sanitize committed response errors

Keep post-write response conversion failures behind a fixed HTTP detail while preserving the retired-pool teardown guarantee.

* fix(mcp): avoid duplicate committed reconciliation error logs

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-10 11:03:26 +08:00
ZJPex
4cb8e3096b
feat(mcp): filter background tasks by status (#6527)
* feat(mcp): filter background tasks by status

* feat(mcp): expose task list filters in Gateway

* docs(mcp): describe background task query filters

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-10 08:53:51 +08:00
spud
db1e0e10b4
fix(mcp): selectively reconcile changed deployment servers (#6487)
* fix(mcp): selectively reconcile changed deployment servers

* fix(mcp): validate the frozen task snapshot before installing epochs

Durable background calls resolve their connection from the configuration frozen
at Gateway startup, so installing a newer revision's binding epoch fenced every
status/cancel call with `StaleMCPBindingError` — even though hot reload of that
server is correctly rejected. The rejection only landed in `get_mcp_tools()`,
which runs after the cache had already installed the epoch.

Both decisions that install an epoch now consult the frozen snapshot first. The
reconciliation planner falls back to the conservative full reset, where callers
that still run against the startup configuration rebind in the replacement pool,
and never installs the rejected epoch. A discovery claim raises the same
`McpTaskConfigurationError` that `get_mcp_tools()` would, before touching the
pool and without recording the revision as applied. Unrelated non-task servers
still reconcile selectively.

Adds the three requested regressions — a rejected hot change keeps background
task callers working, a rejected revision installs neither epoch nor tombstone
(changed connection and disabled server), and an unrelated server change stays
selective — plus mutation proofs for both new gates.

* fix(mcp): always tear down a pool the claim path retired

`reset_session_pool()` only fences and unlinks the singleton; the owners are
signalled by `close_all_sync()`. The claim ran that teardown *after* its critical
section, so a claim that retired the pool for a new shared-reset generation and
then raised — a revision the frozen durable-task snapshot rejects is the
reachable case — left the retired pool's sessions running without ever receiving
a close signal.

Wrap the claim in `try`/`finally` so the retired pool is torn down on every path,
still outside every lock and after the lock is released. Adds a regression
covering the shared-reset-then-rejected-revision sequence: the error propagates,
the retired pool's owner observes exactly one `__aexit__`, the replacement pool
holds no rejected epoch, and restoring the startup configuration initializes
normally again.

* fix(mcp): deliver retired-pool cleanup before the cache reader waits

A synchronous `get_cached_mcp_tools()` reader can apply a full reconciliation
while a discovery is in flight — an interceptor change and a new shared-reset
generation both qualify, because the applied baseline exists before any cache is
published. `reset_session_pool()` only fences and unlinks the pool, and the
reader then entered its wait/retry branch from inside the condition, skipping the
`close_all_sync()` that signals the retired pool's owners. The replacement cache
published normally while the discarded pool kept a live owner and stdio
subprocess that no later reset could reach.

Hoist the observed waiting generation out of the critical section and deliver the
retirement before every wait/retry path, then re-acquire the condition and wait
on that exact generation so the predicate cannot miss a wakeup. Cleanup stays
outside `_init_condition`, and the early `return _mcp_tools_cache` cannot skip a
retirement because applying a plan always clears the published cache.

Adds both triggers as a parametrised regression asserting the retired pool's
owner observes exactly one `__aexit__` and the registry drains — not merely that
the pool is fenced — plus a mutation proof for the wait-path cleanup.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-09 16:41:29 +08:00
Nan Gao
4bab280250
fix(skills): share skill cache invalidation across Gateway replicas (#6495)
* fix(skills): share skill cache invalidation across Gateway replicas

SkillStorage.load_skills() rescans disk and re-reads extensions_config.json
on every call, but the prompt layer in deerflow/agents/lead_agent/prompt.py
caches the enabled-skills list, the per-(app_config, user_id) entries and
the rendered <skill_system> section per process, and every skill mutation
in routers/skills.py only refreshed the process that handled it. With
several uvicorn workers or several Pods sharing one home volume, a skill
installed, edited, deleted, rolled back or toggled through replica A kept
being offered as the old skill set by replica B until B restarted, and
POST /api/skills/reload had to be sent to every Pod separately.

Reuse the MCP cache-reset mechanism. A new generic helper,
deerflow.config.shared_reset_marker, owns a hidden JSON marker beside the
resolved extensions config (.<config name>.<suffix>.json): publish()
replaces it atomically with a random generation under
extensions_config_write_lock + extensions_config_file_lock, chaining
previous_generation and an optional user_id; SharedResetMarkerTracker
polls it at most once per second (monotonic clock, never blocking behind a
concurrent poller), adopts the current state silently on its first poll,
and classifies a change as user-scoped only when exactly one publication
carrying that user happened since the last poll. deerflow/mcp/cache.py now
builds its .<config>.mcp-cache-reset.json marker through the same helper
with no behavior change.

prompt.py publishes .<config>.skills-cache-reset.json via
publish_skills_cache_reset(user_id=...) and checks the marker on every
cache lookup (global cache, per-config cache, prompt-section LRU, user
scoped entries), calling _invalidate_enabled_skills_cache() or
invalidate_user_skill_cache(). The publishing process remembers its own
generation so it does not retire the caches it just rebuilt. Every
mutation tail in routers/skills.py (install/upload, edit, delete,
rollback, enable/disable, reload) refreshes locally and then publishes the
marker off the event loop; /skills/reload answers scope=shared_config,
or scope=process when no extensions config path resolves, and a marker
publication failure is a 500 so a 200 never means only one process was
refreshed.

Docs: README, backend/docs/API.md, skills/config/mcp/gateway AGENTS.md,
and frontend skills.mdx (en/zh; also fixes the reference to a
non-existent skills/loader.py) describe the shared invalidation and drop
the per-Pod guidance. CHANGELOG entry under Unreleased / Fixed.

* docs(changelog): link the skills cache reset entry to #6495

* fix(skills): drain the reload tail and keep marker failures out of 404

Two review follow-ups on the shared skills cache reset (#6495).

POST /skills/reload awaited the local refresh and the marker publication
unshielded. CancelledError is a BaseException, so a caller that
disconnected between the two slipped past the `except Exception -> 500`
mapping, surfaced no error, and left every peer replica serving the old
skill set -- the one outcome the operator hook for external mount writes
exists to prevent. The refresh and the publication now run as one
`await_drained(_drain_skill_mutation("reload", ...))` tail, the same
discipline the install/edit/delete/rollback/toggle tails already use; the
exception -> 500 mapping and the scope response are unchanged.

Publication errors could surface as 404. resolve_shared_config_path()
only swallows the resolution-time FileNotFoundError; a config directory
that vanishes between resolution and the lock file / atomic replace still
raised FileNotFoundError from the marker write, and the install, edit,
delete and rollback handlers mapped that to HTTP 404 with the raw OS
error text for a skill that had in fact been applied. publish_skills_cache_reset()
now re-raises OSError (and a misconfigured project root's ValueError) as
SkillCacheResetPublishError, a RuntimeError that no handler maps to
404/400, so every handler reports it through its generic 500 branch; the
message names only the exception type, never a server path. The install
handler gains an explicit clause that pins that contract beside its 404
mapping.

Tests: cancelling reload after the refresh still publishes the marker;
a FileNotFoundError from the marker write yields 500 (not 404) for
install, edit, delete and rollback, 500 for the public toggle and the
generic reload error; the wrapper type is not an OSError.
2026-10-08 19:43:04 +08:00
Hyeonsang Cho
89c9f08714
fix(mcp): skip task servers whose discovery failed instead of dropping all tools (#6481)
* fix(mcp): skip task servers whose discovery failed instead of dropping all tools

Per-server discovery replaced a failed or timed-out server with an empty
tool list. For a server with task_toolsets, _configure_task_tools_for_server
then reported submit/status/cancel as missing and raised
McpTaskConfigurationError, which get_mcp_tools re-raises, so the tools of
every healthy server were discarded and the cache was never published:
each agent build rediscovered every server.

load_server_tools now returns None for a failed discovery, distinct from a
server that answered with zero tools, and the wrapping loop skips such
servers before task-toolset validation. A server that answers without its
configured raw tools still raises as a configuration error.

* docs(changelog): reference #6481

* docs(mcp): name the cache-reset endpoint as a task-server recovery path

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-08 14:24:51 +08:00
sxh
be34cc46d0
fix(mcp): fall back when a token response carries an unusable expires_in (#6473)
* fix(mcp): fall back when a token response carries an unusable expires_in

OAuthTokenManager._fetch_token() converted the token endpoint's
expires_in to a timedelta while guarding only TypeError/ValueError. Two
server-controlled inputs raised OverflowError, which escaped the parser and
failed the whole token fetch, so the MCP server never received its
Authorization header:

* int(float('inf')) raises OverflowError. json.loads accepts Infinity, so a
  non-finite expires_in reached it uncaught (NaN happened to be caught).
* int(1e30) succeeds, but timedelta is capped at 999_999_999 days and rejects
  the magnitude with OverflowError. A large JSON integer literal behaves the
  same way.

Catch OverflowError alongside TypeError/ValueError and treat a lifetime beyond
timedelta's own range as unusable, so both fall back to the documented
one-hour default like every other malformed value.

Fixes #6472

* fix(mcp): handle both token expiry overflow stages

---------

Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-08 10:44:10 +08:00
spud
b58aea3101
fix(mcp): fence superseded stdio server bindings (#6435)
* test(mcp): pin stdio binding fencing

* fix(mcp): fence superseded stdio server bindings

* docs(mcp): document stdio binding fence

* fix(mcp): address binding fencing review

* fix(mcp): bind discovery to cache-owned pool
2026-10-08 09:02:47 +08:00
asts
b88d8db5fc
fix(mcp): share rotating OAuth state with durable task calls (#6342)
* fix(mcp): share rotating OAuth state with durable task calls

* refactor(mcp): centralize task server selection

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-06 10:27:44 +08:00
spud
3f8f91d064
fix(mcp): isolate pooled sessions by ownership domain (#6133)
* fix(mcp): isolate pooled sessions by ownership domain

Separate deployment and personal stdio sessions even when they share the same runtime server name and thread scope. Preserve deployment call compatibility while carrying personal ownership through Agent wrappers, durable task calls, and disconnect eviction.

Add focused regression coverage for personal discovery wiring and domain-scoped disconnect cleanup.

* test(mcp): address ownership-domain review feedback

Accept the new wrapper keyword in the prefix-overlap test double and reuse MCPPoolDomain for internal task-caller ownership parameters.

* style(mcp): format ownership-domain changes

* style(mcp): apply repository ruff formatting
2026-10-06 09:08:25 +08:00
PeaceMaker-best
90d466c917
feat(mcp): propagate cache resets across workers (#6126)
* feat(mcp): propagate cache resets across workers

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>

* docs: keep MCP reset guidance within instruction budgets

---------

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-01 10:21:24 +08:00
RongJie G
d642dfdd2f
fix(mcp): reserve notification launches atomically (#5965)
* fix(mcp): reserve notification launches atomically

Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>

* fix(mcp): classify busy notification conflicts by type

Signed-off-by: Willem Jiang <willem.jiang@gmail.com>

---------

Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-30 09:12:11 +08:00
NanHan Qing
f8b69bd84b
fix(mcp): close thread-scoped MCP sessions on thread deletion (#5956)
* fix(mcp): close thread-scoped MCP sessions on thread deletion

Deleting a thread removed its filesystem data, checkpoints, runs, events,
feedback, metadata and browser session, but never closed its persistent MCP
sessions. MCPSessionPool.close_scope() had no production caller at all, so a
recreated thread id under the same user could reuse stale stateful MCP server
state, and the leaked owner task plus subprocess kept holding memory and file
descriptors.

Add MCPSessionPool.close_thread_scope(user_id, thread_id) and call it from the
reserved deletion path under the same best-effort policy as the neighbouring
cleanups. It matches every incarnation of the thread -- the legacy user:thread
scope and any v2:[user, thread, incarnation] scope -- because a deletion cannot
reliably read the current incarnation: the record may predate incarnation
tracking, and a new incarnation can be minted concurrently. close_scope() is
unchanged for callers that already hold one exact key.

Add mcp_scope_belongs_to_thread() to own the matching rule, and hoist the v2:
prefix into a shared constant so the encoder and matcher cannot drift.

Refs #5188

* test(mcp): bind owner identity in thread cleanup checks

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-29 16:53:18 +08:00
Wenchao An
52a3e2564f
fix(mcp): isolate personal connections from deployment configuration (#5966)
* fix(mcp): isolate personal connections from deployment configuration

* fix(mcp): preserve personal task ownership and repair plugin E2E fixtures

* fix(mcp): handle corrupt personal config and cache owned calls

* fix(mcp): pin personal endpoint addresses and handle invalid listings

* fix(mcp): recheck current authority for privileged personal connections
2026-09-28 16:02:34 +08:00
asts
b5c5cdd6bc
fix(mcp): restore task owners for background credential selection (#5870)
* fix(mcp): restore task owners for background credential selection

* fix(mcp): narrow background task owner binding
2026-09-26 15:37:52 +08:00
lakers-abing
3a86278047
fix(mcp): preserve backslash paths during bare-filename rewriting (#5815)
* fix(mcp): preserve backslash paths during bare-filename rewriting

* test(mcp): cover the trailing backslash boundary in bare-filename rewrite

Review noted that treating `\` as a separator also suppresses a rewrite when a
single trailing backslash precedes a newline, which is more often a Markdown
hard break, and that neither the tests nor AGENTS.md covered that position.

Lock the behavior in through the production result converter and document the
tradeoff: a backslash is a legal filename character on POSIX, so forgoing that
one rewrite is preferable to risking a corrupted path reference. Runtime
behavior is unchanged.

---------

Co-authored-by: lakers-abing <236384465+lakers-abing@users.noreply.github.com>
2026-09-25 12:47:33 +08:00
Hyeonsang Cho
174c6f7fd3
fix(mcp): run lazy MCP discovery once when initialization itself raises (#5836)
* fix(mcp): run lazy MCP discovery once when initialization itself raises

get_cached_mcp_tools() wrapped the whole lazy-initialization call in the
`except RuntimeError` that was only meant to catch asyncio.get_event_loop()
raising when no loop is set. run_until_complete() and Future.result() re-raise
the coroutine's own exception, so a RuntimeError from discovery — e.g. the
deliberately re-raised McpTaskConfigurationError — took the same fallback and
ran asyncio.run(initialize_mcp_tools()) a second time, respawning every stdio
server and re-fetching OAuth tokens before giving up. Inside a running loop
that fallback fails with "asyncio.run() cannot be called from a running event
loop", so the logged traceback hid the real cause.

Resolve the loop first, in its own try/except, and drive initialization exactly
once: asyncio.run when no loop is set or the current loop is closed, a worker
thread when the loop is running, run_until_complete otherwise. Any failure is
logged once and returns [] as before.

Regression tests pin one discovery call and the real logged cause with and
without a running loop, and keep both fallbacks (no loop set, closed loop)
covered.

* test(mcp): surface worker-thread failures in the no-loop fallback test

Thread.join() does not propagate exceptions, so a regression of the
asyncio.run fallback failed this test with a bare KeyError: 'result' instead
of the real traceback. Capture sys.exc_info() in the worker and re-raise it
on the main thread before the assertions.
2026-09-25 10:28:28 +08:00
xbzz1018
f83bc05a5f
fix(mcp): resolve generated file paths containing spaces (#5819)
* fix(mcp): resolve generated file paths containing spaces

* fix(mcp): avoid partial rewrites at whitespace suffixes

---------

Co-authored-by: xbzz1018 <186267668+xbzz1018@users.noreply.github.com>
2026-09-24 20:13:18 +08:00
spud
8204dcbb20
fix(mcp): invalidate cache only for effective MCP config changes (#5750)
* fix(mcp): avoid cache resets for unrelated config changes

extensions_config.json carries MCP servers, skills and middleware settings, but the MCP cache treated any whole-file signature change as MCP state change, so toggling a skill retired every cached tool and closed all pooled sessions.

Record a snapshot of the effective MCP slice (enabled servers' parsed config plus mcpInterceptors) and retire MCP state only when that slice changes. Hand the exact config revision to tool discovery so a change during discovery cannot publish tools built from a superseded revision, and add a non-initializing refresh entry point so disabling the last enabled server still converges the cache and pool on the next tool assembly.

* docs(mcp): align cache guide and restore error diagnostics

Review feedback on #5750 flagged three non-blocking issues: the MCP module guide still described the pre-semantic-snapshot invalidation contract; the tools.py sanitization covered every exception in the MCP block, discarding diagnostics for failures that cannot carry resolved credentials; and _is_cache_stale()'s docstring presented it as a pure predicate despite adopting a new file signature.

Narrow the type-only logging to ExtensionsConfig.from_file() and restore logger.exception() for tool caching, pool cleanup and plugin filtering, with a regression test asserting the message and traceback survive. Document the effective-MCP-slice comparison, the stable config handoff to discovery, refresh_mcp_cache_if_active() and the _initialized_without_config handling in the module guide, and note the signature-adoption side effect in the docstring.

* fix(mcp): unify config normalization and verify signatures

Review follow-up on #5750.

- Normalize mcpInterceptors (bare string == single-element list; missing key == empty list) and the type/transport alias through one shared module used by the tools cache, the interceptor builder and the durable-task config snapshot, so equivalent spellings no longer retire tools/sessions or raise McpTaskConfigurationError.
- Treat a resolved-path switch as a signal to re-read the effective MCP slice instead of an invalidation reason: an identical slice adopts the new path and signature and keeps pooled sessions.
- Refuse to publish or adopt a signature whose sha256 digest is missing ((mtime, size, None) from a file that could not be read).
- Sync the English and Chinese MCP user docs with the effective-config comparison, and document that the Gateway admin reset stays unconditional.

* docs(mcp): note the unverifiable-signature fail-closed branch

A present config file whose content cannot be hashed is treated as stale, unlike a deleted file which keeps the last-known-good cache. Record that asymmetry in the module guide.

* docs(mcp): scope the no-hash rationale to never-initialized processes

The cost-free early return in refresh_mcp_cache_if_active() only covers a process that never published an MCP cache. A process that did publish one (including an empty cache from the zero-server or LangGraph Studio lazy-init path) still pays one stat+sha256 per tool assembly. Correct the rationale in the docstring and the matching comment in get_available_tools().
2026-09-24 17:19:01 +08:00
asts
eae70b5e40
fix(mcp): include connection setup in task session timeout (#5733)
* fix(mcp): include connection setup in task session timeout

* fix(mcp): identify remote session initialization timeouts

* style(tests): format MCP timeout assertions

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-23 10:20:51 +08:00
RongJie G
e01314442c
fix(mcp): scope sessions and task access by thread incarnation (#5556)
* fix(mcp): scope sessions and task access by thread incarnation

* fix(mcp): preserve thread incarnation in delegated subagents

* fix(mcp): preserve incarnation in durable batches

* fix(mcp): bind standalone graph lifecycle context

* fix(studio): preserve implicit thread creation metadata

---------

Co-authored-by: CorgiBoyG <CorgiBoyG@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-23 08:33:20 +08:00
asts
0ff2e9ddcf
fix(mcp): honor configured stdio working directories (#5643)
* fix(mcp): honor configured stdio working directories

* fix(mcp): preserve defaults for empty working directories

Treat empty stdio cwd values as omitted, including unresolved environment references. Add real subprocess regressions for discovery and pooled-call defaults, plus direct connection-dictionary coverage.
2026-09-22 11:15:06 +08:00
spud
906c3d4554
fix(mcp): make durable task claims cancellation-safe (#4966)
* feat(mcp): re-scope to MCP task claim lifecycle only

Keep PR #4966 a small, closed MCP lease/cancellation state-machine change and
move RunJournal and Run lifecycle work into dedicated follow-ups. This branch
contains only the MCP task claim lifecycle:

- mcp task release/snapshot fencing by owner + per-claim lease token
- phase-level single-flight poll/cancel/notification owners with retained handoff
- routine cancellation no longer persisted as a task failure diagnostic
- bounded ordinary release ownership retention past the drain deadline
- 0018_mcp_task_lease_tokens migration + migration/bootstrap head assertions
- wait_for_task_until helper (MCP uses it); worker-specific capture helper moved
  to the run-finalization follow-up

RunJournal (journal.py + test_run_journal.py) and run lifecycle
(manager/worker/store/run sql + run tests) are preserved on
backup/cancellation-safety-full and will be raised as separate follow-ups.

* fix(mcp): unblock claims after ambiguous handoff resolves

A phase-level single-flight owner only guards an ambiguous claim outcome. Once
the claim resolves, the phase owner is released immediately; the handoff may
continue releasing returned rows as bounded, service-owned background work
(transferred to _compensation_tasks on timeout). Per-claim token fencing rejects
a late release against a newer claim generation, so a stuck release no longer
locks the whole phase until process restart.

- README: drop the stale progress-snapshot sentence from the bounded ordinary
  release description.
- service: pop the identity-checked phase owner as soon as the claim outcome is
  known, then release returned rows with the bounded path; carry the release in
  _compensation_tasks if it exceeds the drain deadline.
- mcp/AGENTS.md: document that only an unresolved claim outcome (not the handoff)
  blocks later phase scans, and that returned-row releases may continue in the
  background once the owner is released.
- tests: pin that the phase owner is released before a stuck release finishes
  while the release stays service strong-owned.

* refactor(mcp): remove unused single-record claim wrappers

_poll_one, _cancel_one, and _notify_one are unreachable in production: the
worker always processes claimed records through _run_claimed_batch, so these
wrappers preserved a second, dead single-record lifecycle (state is None)
whose only observable behavior was a wrapper-specific cancellation release.

Remove the three wrappers and migrate the regressions that guarded their
cancel/release invariants to exercise the production _run_claimed_batch path
(operation=_*_one_claimed, release=_release_*_after_cancellation). The single
wrapper-only "state is None" contract (test_poll_release_hang_without_batch)
is deleted; all 11 remaining invariants (CancelledError preservation, repeated
cancellation, poll-only token-fenced lease release, notification claimed vs
dispatched phase release, hung compensation -> service ownership, and
background compensation exactly-once observation) are now covered through the
real batch lifecycle.

* fix(mcp): fence claim-owned mutations against stale generations

The per-claim token check in the ORM release/apply paths was only in the
SELECT; the final write went out by primary key. On SQLite (where
with_for_update() is a no-op) a mutation from an older claim generation
could therefore clear a claim that a newer generation had reclaimed after lease
expiry — the exact distributed lease-fencing failure the per-claim token was
meant to prevent.

Make every claim-owned mutation a single atomic conditional UPDATE with the
owner and per-claim token in the WHERE clause (rowcount 0 => stale, return
False, no mutation):

- release_claim: atomic fence; record the poll-failure event after the fence
  wins (same transaction, holding the write lock).
- apply_snapshot / apply_cancel_snapshot: atomic fence; record the event after.
- finish_notification_run: atomic fence; use a CASE on event_version >>
  dispatch_version to keep a newer event pending for redelivery instead of
  swallowing it as delivered.

Add one regression per path: a stale generation's release/apply/finish after a
same-worker reclaim is rejected and never clears the newer claim.

* test(mcp): pin the migration chain head to the lease-token revision

0026_mcp_task_lease_tokens becomes the alembic head, so the chain-head pin in the 0025 repair test had to move on. Follow the 0023 precedent there (single head plus expected predecessor) instead of pinning a literal head, and give the new revision its own migration test, which owns the pin and covers the nullable claim-token columns on upgrade and their removal on downgrade.

* refactor(mcp): close cancellation cleanup leftovers

* fix(mcp): retain cancelled release diagnostics

* test(mcp): remove obsolete settled compensation case

* test(mcp): cover interleaved lease reclaim races

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-20 19:11:42 +08:00
哈基米
796ca28f55
fix(mcp): insert bare-filename rewrites literally (#5522)
_rewrite_unique_bare_filenames handed the correlated /mnt/user-data virtual
path to Pattern.subn as a replacement template. That path is built from the
file's relative path, and a backslash is an ordinary character in a POSIX
filename, so a file written literally as "screenshots\q3.png" -- the shape a
model produces by passing a Windows-style path to a stdio server on a POSIX
host -- turned \q into an unknown template escape. Pattern.subn compiles the
template eagerly, so re.error escaped _convert_call_tool_result and failed the
whole tool call even though the server had already written the file, and the
agent never saw the path.

When the backslash does start a known escape (\r, \t, \b ...), the bare-filename
pass substituted that byte into the returned text instead, so
"screenshots\raw.png" came back as a path with a raw CR in the middle of it.

Insert the correlated path through a callable replacement, matching what
_rewrite_local_paths_in_text already does, so it is never parsed as a template.
2026-09-18 08:03:37 +08:00
yang rui
dfe9a520b9
fix(mcp): isolate pooled sessions by owning event loop (#5396)
* fix(mcp): isolate pooled sessions by owning event loop

* refactor(mcp): remove obsolete eviction cancellation plumbing
2026-09-13 17:13:45 +08:00
XIIRUAN
c65737025b
fix(mcp): enforce session pool capacity during promotion (#4962)
* fix(mcp): enforce session pool capacity during promotion

* test(mcp): cover concurrent session promotion

* docs(mcp): document promotion-time capacity check

* fix(mcp): align capacity eviction with owner promotion

* fix(mcp): detach promotion eviction teardown from new owner

* test(mcp): keep eviction teardown regression focused

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: XIIRUAN <253657638+XIIRUAN@users.noreply.github.com>
2026-09-10 21:21:18 +08:00
Shxiao
cbd6621d52
fix(mcp): resolve drive-qualified paths in file reference rewriting (#5242)
* fix(mcp): resolve drive-qualified paths in file reference rewriting

urlparse reads a Windows drive prefix ("C:/...") as the URI scheme, so
_local_path_from_uri() returned None for every drive-qualified path and
MCP file references were never rewritten to /mnt/user-data/... virtual
paths on Windows hosts. file:// URIs were parsed with urlparse().path
alone, which also drops the drive qualifier.

- resolve file URIs through url2pathname so the /C:/... form keeps its
  drive, and treat single-letter schemes as bare drive paths;
- match drive-qualified absolute paths in the free-text reference regex;
- build test URIs with Path.as_uri() and anchor absolute-path fixtures
  at tmp_path so expectations are host-portable, and cover the
  drive-prefix scheme quirk explicitly.

* fix(mcp): decode file URIs once and guard Windows path rejection

Review follow-up on #5242:

- url2pathname already percent-decodes on both platforms, so the extra
  unquote() wrapper decoded references twice and broke filenames that
  contain a literal '%'. Pass parsed.path straight through.
- On Windows, url2pathname raises OSError for paths containing a raw
  '|' (e.g. file:///C:/tmp/a|b.png); catch it so one odd URI cannot
  abort the whole best-effort rewrite pass.
- The relative-reference regex alternative now accepts backslash
  separators, which is what Windows servers print for relative paths.
- Add Windows-only regressions driving the backslash free-text form
  and a file:///C:/ URI end to end, plus the OSError rejection.

* fix(mcp): resolve file://C:/… URIs with a drive-qualified authority

Review follow-up on #5242 (two-slash Windows drive form):

- Some Windows tools emit file://C:/… without the third slash, which
  puts the drive in the URI authority. Consult parsed.netloc: rebuild
  the /C:/… URL path for a drive-qualified authority, keep the current
  handling for empty and localhost authorities, and reject any other
  host instead of silently treating its path as local.
- Extend the free-text regex so the two-slash form matches as one token
  instead of the previous stray e://… mid-token match.
- Cover the two-slash form at the _local_path_from_uri unit, through
  _rewrite_local_paths_in_text, and add a portable case asserting that
  a remote-host file URI is ignored.

* fix(mcp): anchor the drive-qualified text alternative with a lookbehind

Review follow-up on #5242:

- [A-Za-z]:[\/] could steal a token at an earlier scan position:
  for file:/tmp/… (single-slash form per RFC 8089 / Java File.toURI())
  the match became e:/tmp/…, which resolves as a bare drive path and
  left the reference unrewritten where /tmp/… was rewritten before.
  Anchor the alternative with (?<![\w.-]) so word:/… shapes fall
  through to the earlier alternatives.
- Add the missing coverage for the relative alternative's backslash
  support (temp\page.yml through _rewrite_local_paths_in_text) and
  a portable regression pinning the file:/… tokenization.
2026-09-07 18:43:01 +08:00
goloisme
683d146a30
fix(mcp): MCP cache re-initialization broken by cross-loop asyncio.Lock (#5062)
* Fixes #5060: P1 snapshot config before loading, P2 RLock for sync path

P1: Config changes during initialization can permanently cache stale tools.
    - Snapshot _config_path and _config_signature BEFORE await get_mcp_tools()
    - Compare AFTER get_mcp_tools() completes using _current_config_state()
    - If config changed during loading, discard stale result and retry
    - Prevents publishing old tools with new signature, which would make
      _is_cache_stale() permanently return False

P2: Module-level asyncio.Lock still fails across event loops after real contention.
    - _init_lock = threading.RLock() for sync path (reentrant, prevents races)
    - _async_init_lock = asyncio.Lock() for async init serialization
    - reset_mcp_tools_cache() now acquires _init_lock for serialization

Also fixes test P3: removed duplicated test bodies that leaked state between tests.

* fix(mcp): make cache initialization cross-loop safe

- Replace the module-level asyncio.Lock with thread-safe generation claiming
- Snapshot config state before/after MCP loading and discard stale results
- Keep reset state changes short and non-blocking for async endpoints
- Add regression coverage for contended cross-loop init, config rewrites during load, and reset while init is in flight

* fix: release MCP init claim on cancellation

Release the in-flight generation claim from a cancellation-safe finally block so cancelling the task that owns initialization does not strand future callers. Add regression coverage for cancelling the owner and then reinitializing successfully.

* fix(mcp): retire session pool before cache reset release

Prevent a concurrent MCP cache initializer from publishing tool wrappers
bound to the session-pool singleton that reset_mcp_tools_cache() is
already retiring. Add regression coverage for that interleaving.

* fix(mcp): retire session pool on stale cache invalidation

* fix(mcp): retire pool on init discard

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-04 11:30:36 +08:00
rayhpeng
cd35363a05
fix(history): early user messages vanish or jump mid-run when pagination and context compaction overlap (#4696)
* fix(history): stop dropping user messages that fall outside the loaded page window

Two independent paths made a user's own message disappear from a long thread
(#4666, #4508, #4363). Both are reproduced by a real two-round run: once the
thread passes the 50-row `/messages/page` window AND context compaction fires,
the two sources of truth stop overlapping at the head.

1. Middleware-answered tool results never reached the event store. A middleware
   that short-circuits a tool call (e.g. ReadBeforeWriteMiddleware's blocked
   write) returns a user-visible ToolMessage, but LangChain never emits
   `on_tool_end`, so RunJournal never persisted it — the user saw it during the
   run and it vanished on reload. RunJournal already reconciles final-output
   tool messages, but only for an `ask_clarification` allowlist. The allowlist
   is removed; scope stays bounded by the three conditions that actually matter
   (visible, this run's lead agent, not already persisted), so subagent results
   still stay in their own step feed.

2. mergeMessages discarded the checkpoint prefix before the first shared anchor.
   #4065 correctly established that a summarization-rescued early message must
   not be appended to the tail, and suppressed it instead. That suppression is
   what deletes the message when the first history page no longer reaches back
   to it. It is now woven in before the first shared anchor — the one position
   both the checkpoint and seq-sorted history agree on — so #4065's invariant
   (never the tail) still holds. A collapsed unloaded gap is recoverable by
   paging; a dropped message is not.

Verified against real captured payloads from the reproducing run: the first user
message returns to the transcript. Its exact position is still approximate —
after compaction the live window carries too few anchors to place it precisely,
which only seq-based ordering can close.

Backend: 10809 passed (baseline 10808; same 15 pre-existing failures in
browser/crawler community tools). Frontend: 986 passed, typecheck + eslint clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(events): look up a persisted message's seq by identity

Groundwork for placing checkpoint messages in the seq-ordered thread feed
(#4666). A checkpoint carries no seq of its own and loses messages to
summarization, so once the feed's 50-row page window no longer reaches back to a
surviving old message, a client has nothing to place it by. The seq already
exists in run_events keyed by the message id — this exposes it without paging
the whole feed.

`message_identity` is the backend half of the identity rule the frontend applies
in `hooks.ts::messageIdentity`: a ToolMessage is keyed by `tool_call_id`, and
DynamicContextMiddleware's `X` / `X__user` human copies collapse to one identity.
The two halves must stay in sync — a mismatch is silent, degrading placement
rather than raising.

`get_message_seqs` is implemented for all three stores. Misses are absent from
the result rather than an error, so callers degrade to their own placement rule;
the earliest seq wins when one identity resolves to several rows, so a
re-persisted message keeps the position it first occupied. The DB store decodes
rows in Python because `content` is a TEXT column holding a JSON string, not a
JSON column — the identity fields cannot be projected in SQL.

Nothing consumes this yet; no behavior change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(runtime): carry each persisted message's feed seq on values frames

Attaches `additional_kwargs.deerflow_seq` to messages in a root `values` frame
that the thread feed already holds, so a client can place a message the
checkpoint kept but its loaded history page window no longer reaches (#4666).
Nothing is written back to the checkpoint: the seq is added when the frame is
serialized and belongs to that frame only.

Cost is bounded to frames introducing identities the run has not resolved yet.
Messages this run produces are not in the feed while streaming, so they are
looked up once, recorded as misses, and never retried — in a real run the only
frame that pays for a query is the one where compaction brings older messages
back into view. Measured on a reproducing two-round run: 1 lookup across 25
values frames.

The stamper is built once per run rather than per `_stream_once`, or a goal
continuation would discard the resolved seqs. Subgraph frames are not stamped:
a subagent's snapshot is not part of this thread's feed ordering. A lookup
failure logs and leaves the frame unstamped rather than failing it — placement
is an enhancement and clients fall back to their own ordering rule.

Frontend does not read the field yet; no behavior change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(gateway): strip the server-owned message seq from untrusted input

`deerflow_seq` is display metadata the Gateway attaches when it serializes a
values frame. A client replaying messages (regenerate / edit-and-rerun) would
otherwise write it into the checkpoint, where it becomes wrong the moment the
thread is forked — a branch re-seeds its feed and reassigns seq (#4380).

Joins the existing server-owned key set, so it follows the same trusted-internal
rule as the dynamic-context and view-image markers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(frontend): place a checkpoint message by its feed seq, not its nearest anchor

Completes #4666. Weaving a compaction-rescued message before the first shared
anchor keeps it in the transcript, but not in the right place: after compaction
the live window carries too few anchors, and the nearest one can sit deep inside
the loaded page window — measured at row 25 of 50 on a reproducing run, which is
why the first user turn rendered mid-transcript instead of at the head.

Both sides now carry the backend's thread-global seq. `buildVisibleHistoryMessages`
copies each row's `seq` onto the message (same shape as the existing `run_id`),
and the Gateway stamps it onto `values` frame messages it has already persisted.
A live message whose seq is below the loaded window's lower bound is placed ahead
of everything on screen rather than before the nearest anchor. A message with no
seq — still streaming, so not in the feed yet — keeps the weaving path, since the
tail is already its correct position.

Verified against the captured payloads of the reproducing run: the first user
message goes from absent, to #13 (behind the second question), to #0.

Frontend: 988 passed, typecheck + eslint clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(frontend): place a pre-window checkpoint message even when no anchor is shared

Also #4666. Placing a compaction-rescued message by its feed seq was gated on
reaching a shared anchor, because the split ran inside the anchor walk. When the
loaded page and the live checkpoint share no identity at all, that walk never
runs and the message fell through to `[...canonical, ...live]` — appended after
the entire window, the one arrangement #4065 proved wrong, with its seq known
the whole time.

That is not a corner case. Open an old, already-summarized conversation and send
a message: the page on screen is the newest rows from before that turn, while
the checkpoint holds the rescued first user turn plus steps of the new run that
are not in the feed yet. On a reproducing run the two sides shared zero anchors
and the user's own first question rendered at row 50 of 50 — the reported
"first message jumps to the bottom".

Split `beforeWindow` out of `live` before walking anchors, walk `liveInWindow`,
and use it for the no-anchor branch as well, so a message routed ahead of the
window is not re-appended at the tail by dedup.

Measured on captured payloads of a reproducing run (real gateway, real
compaction), first user message position:

  no shared anchor:  row 50 -> row 0, seq order monotonic again
  shared anchors:    row 0 -> row 0 (unchanged)
  paged to the top:  row 0 -> row 0 (unchanged)

Regression test verified red-green: reverting the fix fails it with the message
rendered after the window.

Frontend: 989 passed, eslint + tsc clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(gateway): stamp the message feed seq on checkpoint reads, not only on stream frames

Completes #4666. `_MessageSeqStamper` sits on the streaming publish path, so a
client that joins a live run learns where a summarization-rescued turn belongs
while a client that merely opens the conversation does not — and opening is the
common case. `GET /threads/{id}/state` and `POST /threads/{id}/history` returned
the checkpoint with no seq at all, so the merge fell back to the nearest shared
anchor, which after summarization sits deep inside the loaded page.

Reproduced in a browser against a real gateway, on a thread that had already
compacted: the user's first question rendered at row 320 of 389, behind the
newest question instead of at the head. Both reads showed 0 of 13 messages
carrying a seq. That is the reported symptom, still present after the streaming
fix.

Add `stamp_messages_with_seq`, the request-scoped counterpart of the stamper:
everything a checkpoint still holds is already persisted, so one batched lookup
resolves the whole list and there is nothing to retry later. Resolve the store
through `_optional_run_event_store` rather than `get_run_event_store`, because
seq is placement metadata — a deployment without a feed must still be able to
read a thread.

After the fix, on the same thread in the same browser: 13 of 13 messages carry a
seq and the first question renders at the head, ahead of the newest one.

Backend: ruff clean, 326 passed across the touched suites.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* refactor(harness): move the injected-user-id suffix helpers to utils.messages to break an import cycle

message_identity imported strip_injected_user_message_id_suffix from the
dynamic-context middleware, closing a cycle (middleware -> deerflow.runtime
-> worker -> events -> middleware) that only stayed hidden while an earlier
import happened to break it. Define INJECTED_USER_MESSAGE_ID_SUFFIX and the
strip helper in deerflow.utils.messages and re-export them from the
middleware so existing importers keep working.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(docs): improve formatting and clarity in AGENTS.md and message-merge.test.ts

* perf(events): stop the seq scan once every wanted identity is resolved

Rows past the last wanted seq can only be re-persisted copies that
already lose the earliest-seq-wins tiebreak, so all three stores now
break out of the scan (and the db store out of its per-row JSON
decoding) once found covers wanted. Matters most for /state and
/history reads of long threads, where this lookup runs with no run
cache and a typically tiny wanted set.

Raised by review on #4696.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(events): share the seq-stamping expression between the two stampers

The walrus-plus-merge expression was duplicated verbatim between
stamp_messages_with_seq and _MessageSeqStamper.stamp — two counterparts
of one rule where silent divergence is the likely failure mode if only
one side is edited. Both now call attach_message_seq next to
MESSAGE_SEQ_KEY in message_identity.py. The trailing
isinstance(message, Mapping) guard was unreachable (a non-Mapping entry
already got identity = None) and is gone with the extraction.

Raised by review on #4696.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(events): seq stamping survives launch paths without user context

The db store's get_message_seqs defaults to user_id=AUTO, which raises
when no user is in the contextvar — the first strict-AUTO read ever
called from the worker context. On a launch path that never inherits
the auth context (e.g. a null-owner scheduled task), stamp()'s except
clause swallowed that into a per-frame warning and silently disabled
seq stamping for exactly the background runs that need it.

The stamper now soft-resolves the user id once at build time — the
same rule as the worker's write paths beside it (unset -> no filter)
— and passes it explicitly. jsonl/memory stores gain the same
user_id kwarg the base list_messages contract already carries.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* perf(events): SQL-prefilter the message seq lookup's candidate rows

get_message_seqs scanned and JSON-decoded every message row of the
thread: the early exit never fires when a wanted identity is absent
from the feed (a message still streaming, or checkpoint-only), and
/state / /history reads want the newest messages, so the ascending
scan traversed essentially the whole feed — with the content column
carrying full tool outputs, that is heavy I/O plus N JSON parses on
exactly the long threads this lookup exists for.

A LIKE prefilter now keeps that cost in SQL: only rows containing a
wanted raw id as a substring are fetched and decoded. False positives
are re-checked by message_identity; LIKE wildcards are escaped; an id
json.dumps would escape (breaking the verbatim-substring guarantee)
falls the whole set back to the full scan rather than silently
missing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(agents): sink runtime mechanism docs below the gateway guidance budget

Merging main pushed backend/app/gateway/AGENTS.md past its 40KB soft
budget (main had left 81 bytes of headroom). Per the nearest-file rule,
move the mechanism detail of the message-seq stamping and run-delivery
receipt sections — both owned by runtime/ code — into
packages/harness/deerflow/runtime/AGENTS.md, leaving the gateway file
the REST-surface summary and a pointer. The seq section also documents
the stamper's build-time soft user-id resolution and the db store's SQL
prefilter from the review follow-ups.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(agents): sink durable-MCP task detail below the backend guidance budget

Merging main pushed backend/AGENTS.md past its 24KB module soft budget
(main itself is at 24762 after #4848 — this branch adds zero net bytes
to the file). Per the nearest-file rule, move the two durable-MCP task
runtime bullets' mechanism detail into
packages/harness/deerflow/mcp/AGENTS.md, leaving summaries and
pointers; this also restores ~2KB of headroom so the next merge does
not trip the same wire.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(events): re-ask a message-seq miss once the feed advances

The run-scoped stamper cached lookup misses for the whole run. A message
this run produces reaches a values frame before RunJournal flushes it, so
its first lookup legitimately misses — and the journal persists it moments
later, giving it a feed seq the stamper never asks for again. A long run
that afterwards rolls past the history page and compacts then carries that
message unstamped, back to the approximate anchor placement this stamper
exists to replace (#4666). A transient store error had the same permanent
effect, since the except clause degrades to an empty result.

A miss is now provisional while a hit stays final: RunJournal counts its
successful event-store writes as `feed_generation`, and the stamper re-asks
a missed identity only once that counter moves. Retrying is therefore
bounded by feed writes rather than by frames — the per-frame query the
run-scoped cache was built to avoid — and a failed lookup costs one
generation instead of the run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 22:04:17 +08:00
Terminator666666
c17aa8b98f
fix(mcp): reject credentials that cannot travel as HTTP header values (#5066)
* fix(mcp): reject credentials that cannot travel as HTTP header values

A request-scoped secret or user_auth credential with a trailing newline
(the usual result of reading a token from a file, or a CRLF env-file),
CR/LF, surrounding whitespace, or characters outside Latin-1 sailed
through the credential interceptors into the HTTP client, where httpx/h11
reject it with an exception that echoes the full value:

    LocalProtocolError: Illegal header value b'Bearer sk-...\n'

ToolErrorHandlingMiddleware copies that message into a model-visible
ToolMessage, so the secret landed in the prompt, the checkpoint, and
traces - everywhere headers_from_context promises it never goes.

Add illegal_header_value_reason to mcp/headers.py, mirroring the
transport's own rules (Latin-1 encodable; h11's field_vchar is [^\x00\s]
with SP/HTAB legal only between visible characters), and fail closed in
both interceptors before the value can reach the client. The denial names
only the secret key (plus the reason) and never repeats the value.

Illegal values are denied regardless of on_missing: the key is present,
so a passthrough fallback would silently run the call under the shared
discovery credential - the exact authority confusion the deny default
exists to prevent.

Values the transport accepts are not rejected: embedded SP/HTAB
('Bearer <token>'), Latin-1 high bytes, and DEL all still pass, pinned
by tests against h11's observed behaviour.

* fix(mcp): tighten header value validation to httpx's ASCII boundary

The validator mirrored h11's Latin-1 boundary, but the transport rejects
more than h11 does: build_server_params hands dict[str, str] headers
through the MCP SDK's create_mcp_http_client into httpx.AsyncClient, and
httpx (pinned 0.28.1) encodes str header values as ASCII - so a Latin-1
high byte like 'Bearer caf\xe9' passed validation here only to raise
UnicodeEncodeError inside httpx before h11 ever ran, with the exception
message repeating the offending value.

Validate str values against ASCII instead, flip the tests that pinned
Latin-1 high bytes as transportable, and pin the boundary against the
real client: create_mcp_http_client must reject what the validator
flags and construct cleanly for what it accepts (embedded SP/HTAB and
DEL still pass).

Addresses review feedback on the ASCII vs Latin-1 boundary.

* fix(mcp): validate OAuth and static header values at the same boundary

The validator added for headers_from_context and user_auth left two paths
uncovered. A token endpoint returning an access_token or token_type with a
newline reached httpx/h11, which raise with the full token in the message, and
ToolErrorHandlingMiddleware copies that message into a model-visible
ToolMessage -- the leak this PR set out to close. The operator's static headers
had the same hole.

OAuthTokenManager.get_authorization_header now renders the Authorization value
through one checked helper, so the tool interceptor, the initial discovery
headers and the durable task path are all covered by a single guard. The
rendered value is what gets checked rather than the two fields separately,
because that is what the transport sees: an access_token with leading
whitespace is legal once it follows "Bearer ".

build_server_params applies the same check to statically configured headers.
build_servers_config already isolates a per-server failure, so a bad value
drops that one server and logs the reason instead of the value.

* docs(mcp): correct which transport echoes the full header value

The rationale claimed httpx and h11 both render the full value into their
exception message. Only h11 does, on the line break and surrounding whitespace
cases. httpx's ASCII failure is a UnicodeEncodeError naming the offending
character and its position, not the credential, so at most one character
escapes there; refusing the value up front buys an actionable error rather than
an encode failure raised from inside the client.

Corrected in headers.py and in every copy of the claim: context_headers.py,
user_scoped_auth.py, oauth.py, client.py, mcp/AGENTS.md, docs/MCP_SERVER.md,
the frontend mcp.mdx, and the test comments carrying the same wording. No
behavior change.

---------

Co-authored-by: Terminator666666 <Terminator666666@users.noreply.github.com>
2026-08-31 15:07:30 +08:00
hataa
567a06783c
fix(mcp): tear down the in-flight owner when get_session is cancelled mid-eviction (#5008)
* fix(mcp): keep session owner teardown safe across cancellation paths

* fix(mcp): gate pooled-session publication on the commit inside the owner task

The owner resolved `ready` as soon as initialize() finished, but the
session only became pool property when the creator promoted it into
_entries in Phase 4. A concurrent get_session() could join the in-flight
creation and receive that session from Phase 2b while the creator was
still parked in the Phase-2 eviction teardown; cancelling the creator
then ran the Phase-2 unwind, which unconditionally shut the owner down —
closing the session underneath the joiner (#5008 review).

Move the commit into the owner task: initialize() success now pops the
in-flight record, registers the session in _entries, and resolves ready
with the session in one atomic critical section, so 'ready resolved with
a result' is exactly 'session registered and pool-owned'. Joiners can
therefore only ever receive a committed session, and both creator unwind
paths (Phase 2 and Phase 3) skip teardown when the creation already
committed, leaving the pooled session to LRU eviction / close_*. When the
record was removed before the commit (close_* or creator unwind), the
owner aborts and ready carries the same cancellation the old Phase-4
not-still-ours path raised, so joiners fail with the creation's outcome
instead of hanging or holding an unmanaged session.

test_cancelled_creator_does_not_close_session_held_by_joiner reproduces
the review's scenario deterministically (MAX_SESSIONS=1, hung LRU victim,
gated initialize, second caller receives the session, creator cancelled):
red on the previous commit, green now.
test_joiner_follows_creation_outcome_when_creator_is_cancelled pins the
joiner outcome-gating semantics as a drift guard.
2026-08-30 11:36:26 +08:00