mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-30 23:20:35 +00:00
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
asts
|
64dab5897a
|
fix(mcp): preserve remote URLs during local file rewriting (#6878)
* fix(mcp): preserve remote URLs during local file rewriting * fix(mcp): honor escaped delimiters in remote URLs |
||
|
Storm
|
588ad010c0
|
fix(mcp): stop persisting URL-sourced ResourceLink blocks and heal bricked threads (#6606)
* fix(agents): sanitize persisted URL content blocks before model calls
Threads whose checkpoints carry URL-sourced file blocks (written before
the MCP conversion-layer fix) fail every model call at serialization: the
Chat Completions translator rejects any file block carrying a url, and the
error handler answers each later turn with a non-retriable fallback. Add a
read-time sanitizer that rewrites those blocks in the model-bound request
view only, so poisoned threads heal on their next turn with no migration
and no user action.
- ModelContentCompatibilityMiddleware wraps the model call (sync and
async) and rewrites ToolMessage list content: URL-sourced file blocks
become "[Resource ({mime or unknown type}) available at {url}]" text
blocks, and image blocks whose scheme is not http/https/data get the
same placeholder. Fetchable image URLs, base64 payloads, text, and
non-ToolMessage messages pass through. The rewrite is order-preserving,
idempotent, and never writes state or mutates checkpoints, so artifact
capture and other state readers keep seeing the original blocks.
- Register on all three chains that replay checkpointed ToolMessages:
build_middlewares (lead; outer of the MODEL_PHYSICAL extension anchor so
its final-request guarantee holds), the SDK factory
_assemble_from_features (new step 13a; docstring updated and its stale
omission of TokenBudgetMiddleware fixed), and
build_subagent_runtime_middlewares (after DurableContext/summarization,
before the date/coalescer tail).
- Subagent coverage verified by tracing every subagent agent-creation
site: executor.py _create_agent is the only caller of
build_subagent_runtime_middlewares and serves both the task tool and the
durable batch worker (SubagentRuntime itself only carries dependencies),
and a wiring test now asserts the middleware is present in that
builder's output; the lead and factory chains gained the same wiring
assertions.
- Update the chain-order contract tests for the new slot: full-chain
order, loop-detection adjacency, custom loop-detection tail positions,
and the subagent chain count (22 -> 23).
* fix(mcp): stop emitting URL-sourced file blocks from ResourceLink results
MCP ResourceLink conversion produced a URL-sourced file content block for
every non-image link, which LangChain's Chat Completions translator rejects
at serialization time. Once such a block was checkpointed in the
ToolMessage, every subsequent turn of the thread failed at model-call time,
permanently bricking the conversation.
- Downgrade every ResourceLink that is not an http(s) image to a short text
placeholder carrying the resource name, MIME type, and the
already-virtualized URL, so host paths never leak and the model still
learns a resource was produced and where it lives.
- Preserve each downgraded link as structured data in the tool-message
artifact under resource_links (merged with structured_content when
present) for artifact capture and future MCP Apps rendering.
- Keep http(s) image links as image blocks; embedded base64 payloads, text
path rewriting, and the error-to-ToolException path are unchanged.
- Re-contract the file-migration tests to the new behavior and document
the conversion contract in the MCP AGENTS.md.
* docs(middlewares): add ModelContentCompatibilityMiddleware to the chain inventory
Insert the read-time sanitizer at its actual lead-chain position (after the
token-budget guard, before the custom-middleware injection point) and renumber
the tail entries 34-39 to 35-40.
* fix(agents): rewrite file blocks on url key presence, not truthiness
langchain-core's OpenAI translator raises for ANY file block carrying a
url key — an empty string or None included — so the read-time sanitizer
must not let such blocks through just because the value is falsy. A
persisted block with url="" or url=None would still brick every later
model call in the thread.
- Gate the file-block rewrite on "url" in block instead of a truthiness
check; a non-empty string URL keeps the
"[Resource (mime) available at url]" placeholder, while an empty or
None URL gets a location-less "[Resource (mime)]" placeholder, so no
unusable location is advertised to the model.
- Pin both shapes with new middleware tests (url="" and url=None file
blocks are both rewritten to text). The image branch is unchanged: an
empty-image case fails provider-side with a 400, not a bricked thread.
* fix(mcp): keep data URIs out of model text and resource_links
A data: scheme ResourceLink fell into the generic placeholder branch,
inlining megabytes of base64 into the model-visible text and into
resource_links, which is checkpointed state — against the repo policy
that data:/blob: URIs never enter state.
- Special-case data: links in _convert_call_tool_result: the
model-visible placeholder becomes "[Resource: name (mime) embedded
inline]" and the raw URI is not added to resource_links at all, so the
payload can neither bloat the context window nor persist in the
checkpoint.
- Pin the behavior with a conversion test asserting both the placeholder
wording and the absence of the payload/URI from every emitted block and
the artifact.
- Add the missing passthrough pin for an HTTPS:// uppercase-scheme image
link: scheme matching is case-insensitive, so the link stays an image
block (the block's URL validation normalizes the scheme to lowercase).
* fix(tools): consume resource_links in extract_artifacts_from_result
The MCP conversion layer preserves downgraded ResourceLinks as
{"name", "uri", "mime_type"} entries under the tool message
artifact's resource_links key, but the artifact extractor only read
structured_content — a result whose only artifact signal was
resource_links produced no entries and the file silently vanished from
the artifacts panel.
- Add a fourth extraction path: each resource_links entry becomes a file
artifact entry, reusing _is_referenceable_url like the
structured-content and content-block paths, so data:/blob: and other
unresolvable URIs stay rejected per existing policy. The entry is named
after the link when a name is present, falling back to the URI's final
segment; a string mime_type is carried through.
- Integration-test a result whose only signal is resource_links, plus the
referenceability gate (data:, ui://, and absolute-path URIs in one
artifact).
* docs(mcp): update the ResourceLink conversion contract bullet
Mention the new artifact-registry consumer and the data: special case so
the bullet stays accurate: uppercase http(s) schemes match
case-insensitively, data: URIs get the embedded-inline placeholder and
are dropped from resource_links, and extract_artifacts_from_result turns
resource_links into registry entries through the shared
referenceability gate.
* refactor(tools): add the canonical resource placeholder formatter
Both the MCP conversion layer and the read-time
ModelContentCompatibilityMiddleware downgrade model-incompatible resource
references to a text placeholder, but each built its own literal. Extract
the single formatter they will share so a thread healed at read time shows
the model the same placeholder shape a fresh conversion produces; the name
segment is optional because persisted content blocks carry no resource name.
* fix(mcp): keep host paths out of ResourceLink placeholder text
Review follow-up on the ResourceLink conversion contract. The placeholder
embedded the resolved URL unconditionally, so a link outside the user-data
tree (or converted without thread context) leaked a raw host path — a
file:// URI or bare filesystem path — into model-visible text, violating
US-16. Compute the resolved URL's scheme and withhold the location segment
when it spells out a host filesystem location (empty scheme, file, or a
single-letter Windows drive scheme); virtual paths and remote schemes keep
their location. The unresolved URI still lands in the artifact channel,
which the model never sees.
- add _model_visible_link_location gate and route the generic placeholder
through it, re-contracting the outside-user-data and no-context tests to
the location-less shape and pinning the Windows-drive case
- treat blob: like data: as non-referenceable inline input: drop its raw
URI from checkpointed resource_links and emit the location-less
placeholder instead of the mismatched 'embedded inline' wording, and fix
the comment that claimed blob: was already excluded
- replace every inline placeholder f-string in the ResourceLink branch
with the shared resource_placeholder_text formatter so the conversion
layer and the read-time middleware emit byte-identical shapes; data:
links now render in the canonical location-less form
- sync the ResourceLink conversion contract bullet in mcp/AGENTS.md and
the _convert_call_tool_result docstring with the host-path suppression
rule and blob: handling
* fix(agents): sanitize URL content blocks on any message role
Address the second-pass review of the request sanitizer with two
changes that share the same code path.
- Adopt the canonical resource_placeholder_text formatter from
deerflow.tools.resource_placeholder instead of the module-local
placeholder helpers, so the read-time rewrite and the MCP conversion
layer emit byte-identical placeholder text from one definition.
- Widen the rewrite from ToolMessage-only to every request message with
list-form content. User uploads are verified to produce base64 blocks,
but imported or cross-client history can carry a URL-sourced file
block on a HumanMessage or AIMessage, and the Chat Completions
serializer rejects such a block regardless of the role carrying it.
The rewrite stays request-view-only: state and checkpoints are never
mutated, and base64/file-id blocks and string content pass through
untouched on any role.
* docs(agents): state the ModelContentCompatibility placement contract
The comment above the registration claimed the sanitizer sits just
before Clarification. In the lead chain it is actually followed by the
custom/extension injection point and the whole terminal tail
(terminal-response, finish-reason, and the always-last
ClarificationMiddleware). Describe the real contract: after every
built-in content transform, before extension injection and the
terminal tail.
* docs(middlewares): clarify provenance stamps for request-view-only rewrites
The message-provenance rule read as if every rewrite needs a
provenance_kwargs stamp. Add one line noting the exception: rewrites
that only modify the model-bound request view and never enter state
need no stamps, citing InputSanitizationMiddleware,
DanglingToolCallMiddleware, and ModelContentCompatibilityMiddleware as
precedents.
* fix(agents): keep host paths and data URIs out of healed placeholders
The read-time ModelContentCompatibilityMiddleware rewrote URL-sourced
file blocks and non-fetchable image blocks into placeholders that
inlined the block's URL verbatim. Pre-fix checkpoints — exactly the
threads this middleware exists to heal — can carry a file:// URI, a bare
or Windows host path (links that never resolved to virtual paths), or a
data: URI embedding megabytes of base64, so healing leaked deployment
filesystem layout into prompts (US-16) and could inject the whole inline
payload into model-visible text.
The conversion layer already suppresses those forms via
_model_visible_link_location. Move that gate into the shared
deerflow.tools.resource_placeholder module as model_visible_location
(now also collapsing data:/blob: URIs) and route both middleware
branches through it, so both layers apply identical suppression rules
with one definition. Conversion behavior is unchanged: its data:/blob:
branch short-circuits before the gate, and the gate covers those schemes
for defense in depth.
- resource_placeholder.py gains model_visible_location alongside the
shared formatter; mcp/tools.py drops its private copy and imports it.
- The middleware's file and image branches pass the URL through the gate
before building the placeholder; module docstring documents the rules.
- Tests: gate unit tests (virtual/remote visible; file://, bare/Windows
host paths, data:/blob: withheld) and middleware regression tests
pinning that a pre-fix data: payload, file:// URI, and Windows drive
path heal to location-less placeholders.
- AGENTS.md bullets in mcp/ and middlewares/ state the shared gate.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(agents): downgrade malformed image URLs instead of raising in the sanitizer
The read-time ModelContentCompatibilityMiddleware gated persisted image
blocks on urlparse(url).scheme, but urlparse raises ValueError on
structurally invalid URLs — a bad IPv6 literal, invalid netloc
characters — instead of returning an unusable scheme. Pre-fix
checkpoints and imported history can carry exactly those URLs, and the
block stays in the checkpoint, so every later model call in the thread
raised again: the sanitizer re-bricked the very threads it exists to
heal, only with a framework-side exception instead of the langchain
serialization error.
Catch ValueError and treat the URL as non-fetchable, so the block
degrades into the same text placeholder as any other unresolvable
scheme (the shared model_visible_location gate already swallows
ValueError, so the location segment is withheld as well).
- wrap the image branch's urlparse call in try/except ValueError and
fall back to an empty scheme, keeping http/https/data pass-through
behavior unchanged for well-formed URLs
- add a regression test pinning that an image block with a bad IPv6
literal heals to the location-less placeholder without raising
Co-Authored-By: Codex <noreply@openai.com>
* fix(tools): dedupe artifact refs across extraction sources
One downgraded ResourceLink produced two registry entries: the typed
resource_links channel added an entry for the URI, and the free-text
scan over the conversion layer's placeholder text — which embeds the
same virtual path or remote URL — matched it again. The two handles
pointed at one resource, spent double of the render budget, and showed
the model two handles for a single artifact.
extract_artifacts_from_result now tracks seen real_refs across all
sources and keeps only the first entry per ref. Extraction order makes
the typed sources (structured content, resource_links, content blocks)
win over the free-text scan, so the surviving entry keeps the link's
display name and mime type.
- route all six entry-producing sites through a local _add_unique
helper keyed on real_ref; per-source behavior is unchanged
- document the cross-source dedup in the function docstring
- add a regression test covering a virtual-path and a remote-URL
placeholder together with their resource_links entries
Co-Authored-By: Codex <noreply@openai.com>
* docs(mcp): state the intentional data: image downgrade asymmetry
The conversion layer downgrades image/* data: ResourceLinks to
location-less text placeholders, while the read-time middleware lets
persisted data: image blocks pass through to the model. The two
contracts looked contradictory but are deliberately layered: the
conversion layer keeps new inline payloads out of checkpointed state,
and the middleware only preserves blocks already occupying one.
- mcp/AGENTS.md's ResourceLink conversion contract and the conversion
comment in mcp/tools.py now state the image/* data: downgrade and
why it differs from the read-time pass-through
- middlewares/AGENTS.md's ModelContentCompatibilityMiddleware entry
states the symmetric read-time rule
- pin the behavior with an image-mime data: ResourceLink conversion
test (location-less placeholder, nothing in the artifact channel)
Co-Authored-By: Codex <noreply@openai.com>
* docs(backend): compact AGENTS.md guidance to fit the chain byte budget
The branch's new contract bullets pushed the middlewares and mcp
effective AGENTS.md chains past the 98304-byte AG002 hard limit (the
sandbox/subagents chains on main were already over it). Compact the
guidance without dropping contracts:
- middlewares item 34 points at the MCP ResourceLink conversion
contract and the module docstring instead of restating both
- the mcp ResourceLink conversion contract bullet is tightened
- backend/AGENTS.md merges the File Upload and Context Summarization
bullet sprawl, folds the stray ATX test note into the upload outline
rule, drops the Nginx routing block the root guide already covers,
merges the duplicate MCP-task pointer bullet, compacts Vision
Support, and scopes the Project Structure tree to the backend
subtree (the root guide maps the full repo)
Co-Authored-By: Codex <noreply@openai.com>
* fix(agents): gate the file-URL downgrade on provider support and neutralize placeholder metadata
Two review findings on the read-time sanitizer:
- The file-block downgrade ran for every provider, but only Chat
Completions rejects URL-sourced file blocks: the OpenAI Responses API
(input_file file_url) and Anthropic (document url source) accept
http(s) document URLs, so a persisted block created for those
providers lost its document from the model request. Keep the block
when the selected model serializes http(s) file URLs natively; every
other URL shape (non-http schemes, empty/None, malformed) still
degrades to the placeholder.
- The placeholder interpolated persisted MIME/location metadata
verbatim, but the middleware runs after the input/tool-result
sanitization passes and historical URL blocks never went through them
(non-text content is left untouched), so values carrying
<system-reminder>-style tags or user-input boundary markers reached
the model raw. Route the interpolated metadata through the shared
neutralize_untrusted_tags guard in both the file and image branches.
Tests: provider gating (Responses/Anthropic preserve, Chat Completions
still downgrades, non-http and malformed URLs downgrade even for
capable providers) and metadata neutralization (injection tags
HTML-escaped, boundary tokens neutralized, file and image branches).
Co-Authored-By: Codex <noreply@openai.com>
* fix(tools): require a directory boundary before exposing virtual locations
model_visible_location admitted any URL starting with the virtual
prefix, so a sibling like /mnt/user-data-backups/ops/private/report.pdf
was treated as a virtual path and its host location reached
model-visible placeholder text — defeating the US-16 host-path
suppression the gate exists for. Require the exact virtual root or a
slash-delimited descendant; siblings now collapse to the location-less
placeholder like any other host path. Both the file and image
downgrade branches share the gate, and the conversion layer applies
the same rules to fresh results.
Regressions: gate unit tests (exact root and descendants stay visible;
/mnt/user-data-backups and /mnt/user-database siblings withheld) and
middleware tests pinning the location-less placeholder for both
branches.
Co-Authored-By: Codex <noreply@openai.com>
* docs(agents): scope the placeholder artifact-visibility claim to the conversion layer
The module docstring claimed the healed placeholder keeps the location
visible to the artifact free-text scan and that the link survives in
the structured artifact channel — true for the conversion layer, whose
placeholders are checkpointed, but not on the healing path: the request
view never reaches artifact capture, and no resource_links entry is
created for a downgraded block. Spell out what actually happens here:
a referenceable original block (http(s) URL or absolute host path) is
still captured, from the capture-time scan of the state block rather
than the placeholder, while a non-referenceable one (file://, data:,
blob:) yields no entry and survives only in the untouched checkpoint.
Co-Authored-By: Codex <noreply@openai.com>
* docs(mcp): clarify healing and artifact preservation contracts
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Jason Cat
|
19befe76a6
|
fix(tests): stop hardcoding C: in the Windows file-URI migration tests (#6541)
The two Windows-only cases in tests/test_mcp_file_migration.py asserted a literal "file:///C:/" / "file://C:" prefix on a path built from the pytest temp dir. On a host whose checkout or temp dir lives on another drive both assertions fail, so the only coverage of the drive-qualified file-URI forms handled by _local_uri_to_virtual_path and _rewrite_local_paths_in_text never ran there. Derive the expected drive from the fixture path instead. The URI forms under test are unchanged, so coverage holds on any drive. Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
lakers-abing
|
3a86278047
|
fix(mcp): preserve backslash paths during bare-filename rewriting (#5815)
* fix(mcp): preserve backslash paths during bare-filename rewriting * test(mcp): cover the trailing backslash boundary in bare-filename rewrite Review noted that treating `\` as a separator also suppresses a rewrite when a single trailing backslash precedes a newline, which is more often a Markdown hard break, and that neither the tests nor AGENTS.md covered that position. Lock the behavior in through the production result converter and document the tradeoff: a backslash is a legal filename character on POSIX, so forgoing that one rewrite is preferable to risking a corrupted path reference. Runtime behavior is unchanged. --------- Co-authored-by: lakers-abing <236384465+lakers-abing@users.noreply.github.com> |
||
|
xbzz1018
|
f83bc05a5f
|
fix(mcp): resolve generated file paths containing spaces (#5819)
* fix(mcp): resolve generated file paths containing spaces * fix(mcp): avoid partial rewrites at whitespace suffixes --------- Co-authored-by: xbzz1018 <186267668+xbzz1018@users.noreply.github.com> |
||
|
哈基米
|
796ca28f55
|
fix(mcp): insert bare-filename rewrites literally (#5522)
_rewrite_unique_bare_filenames handed the correlated /mnt/user-data virtual path to Pattern.subn as a replacement template. That path is built from the file's relative path, and a backslash is an ordinary character in a POSIX filename, so a file written literally as "screenshots\q3.png" -- the shape a model produces by passing a Windows-style path to a stdio server on a POSIX host -- turned \q into an unknown template escape. Pattern.subn compiles the template eagerly, so re.error escaped _convert_call_tool_result and failed the whole tool call even though the server had already written the file, and the agent never saw the path. When the backslash does start a known escape (\r, \t, \b ...), the bare-filename pass substituted that byte into the returned text instead, so "screenshots\raw.png" came back as a path with a raw CR in the middle of it. Insert the correlated path through a callable replacement, matching what _rewrite_local_paths_in_text already does, so it is never parsed as a template. |
||
|
Shxiao
|
cbd6621d52
|
fix(mcp): resolve drive-qualified paths in file reference rewriting (#5242)
* fix(mcp): resolve drive-qualified paths in file reference rewriting
urlparse reads a Windows drive prefix ("C:/...") as the URI scheme, so
_local_path_from_uri() returned None for every drive-qualified path and
MCP file references were never rewritten to /mnt/user-data/... virtual
paths on Windows hosts. file:// URIs were parsed with urlparse().path
alone, which also drops the drive qualifier.
- resolve file URIs through url2pathname so the /C:/... form keeps its
drive, and treat single-letter schemes as bare drive paths;
- match drive-qualified absolute paths in the free-text reference regex;
- build test URIs with Path.as_uri() and anchor absolute-path fixtures
at tmp_path so expectations are host-portable, and cover the
drive-prefix scheme quirk explicitly.
* fix(mcp): decode file URIs once and guard Windows path rejection
Review follow-up on #5242:
- url2pathname already percent-decodes on both platforms, so the extra
unquote() wrapper decoded references twice and broke filenames that
contain a literal '%'. Pass parsed.path straight through.
- On Windows, url2pathname raises OSError for paths containing a raw
'|' (e.g. file:///C:/tmp/a|b.png); catch it so one odd URI cannot
abort the whole best-effort rewrite pass.
- The relative-reference regex alternative now accepts backslash
separators, which is what Windows servers print for relative paths.
- Add Windows-only regressions driving the backslash free-text form
and a file:///C:/ URI end to end, plus the OSError rejection.
* fix(mcp): resolve file://C:/… URIs with a drive-qualified authority
Review follow-up on #5242 (two-slash Windows drive form):
- Some Windows tools emit file://C:/… without the third slash, which
puts the drive in the URI authority. Consult parsed.netloc: rebuild
the /C:/… URL path for a drive-qualified authority, keep the current
handling for empty and localhost authorities, and reject any other
host instead of silently treating its path as local.
- Extend the free-text regex so the two-slash form matches as one token
instead of the previous stray e://… mid-token match.
- Cover the two-slash form at the _local_path_from_uri unit, through
_rewrite_local_paths_in_text, and add a portable case asserting that
a remote-host file URI is ignored.
* fix(mcp): anchor the drive-qualified text alternative with a lookbehind
Review follow-up on #5242:
- [A-Za-z]:[\/] could steal a token at an earlier scan position:
for file:/tmp/… (single-slash form per RFC 8089 / Java File.toURI())
the match became e:/tmp/…, which resolves as a bare drive path and
left the reference unrewritten where /tmp/… was rewritten before.
Anchor the alternative with (?<![\w.-]) so word:/… shapes fall
through to the earlier alternatives.
- Add the missing coverage for the relative alternative's backslash
support (temp\page.yml through _rewrite_local_paths_in_text) and
a portable regression pinning the file:/… tokenization.
|
||
|
Nan Gao
|
a5acc25de6
|
fix(mcp): exclude internal temp files from workspace changes (#4898)
* fix(mcp): exclude internal temp files from workspace changes * fix(mcp): address review — shared tmp subdir constant, any-depth docs, nested test - Export MCP_TMP_SUBDIR from constants.py and import it in both stdio launch paths (tools.py, task_tool_caller.py) so the "/tmp" suffix is composed once. - Document that the .mcp exclusion matches by directory name at any depth (like .git/node_modules) in README.md and mcp/AGENTS.md — subagent work dirs below the workspace root get their own .mcp/tmp. - Pin the any-depth semantic in test_workspace_changes.py with a nested workspace/project/.mcp assertion. * docs(mcp): correct any-depth exclusion rationale; re-home tmp pinning comment The previous commit justified the any-depth `.mcp` exclusion with subagent work dirs sitting below the workspace root — a mechanism that doesn't exist: subagents share the parent's thread_id and both stdio launch paths resolve sandbox_work_dir(thread_id), so `.mcp/tmp` is always pinned at the workspace root. Reword mcp/AGENTS.md and the test comment to the real justification (consistency with the other reserved dir names, robustness against a server creating a relative `.mcp` from another cwd). Also move the orphaned "pinning the process temp dir" rationale from tools.py to constants.py next to MCP_TMP_SUBDIR, where both importers see it. |
||
|
yjchen101
|
6fe6bad001
|
fix(mcp): ignore oversized path-like text (#4582)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
VectorPeak
|
07d8b98864
|
fix(mcp): ignore malformed path-like text (#4456)
Co-authored-by: chatgpt-codex-connector[bot] <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com> |
||
|
Zhipeng Zheng
|
53a80d3ad1
|
feat(skills): per-user custom skill isolation with sandbox mounting (#3889)
* feat(skills): per-user skill isolation (#2905) Implement user-scoped skill storage that isolates custom skills between users while sharing public skills globally. Key changes: - Add UserScopedSkillStorage class for per-user custom skill directories - Introduce get_or_new_user_skill_storage() factory with user_id context - Auth middleware sets effective_user_id for request-scoped storage - Agent/prompt/middleware now use user-scoped storage and prompt cache - Sandbox mounts user-scoped skill directories for search/read tools - Add validate_skill_file_path() to SkillStorage for path security - Migration script supports --all-users bulk migration - Frontend: add editable field to Skill type, error check in enableSkill - All skill categories can be toggled (custom skills default to enabled) - Update skill-creator SKILL.md with isolation-aware instructions Tests: - Add test_user_scoped_skill_storage.py (new) - Update all existing skill tests for user-scoped storage - Update sandbox, client, and router tests * fix(skills): address second-round PR review feedback (#3889) - P1-1: restrict legacy skill mount to users without custom skills - P1-2: fail-closed for _is_disabled_skill_path (OSError → return True) - P2-1: AND-merge global extensions_config skill disabled state - P2-2: atomic write for _skill_states.json (mkstemp + replace) - P2-3: normalize X-DeerFlow-Owner-User-Id in trusted boundary - P2-4: LRU-bounded _enabled_skills_by_config_cache (OrderedDict, maxsize=256) - P2-5: clear global prompt cache on PUBLIC skill toggle - P2-6: invalidate skill caches on client.update_skill * fix(tests): correct tool policy test after merge * fix(skills): use DEFAULT_SKILLS_CONTAINER_PATH in UserScopedSkillStorage The "/mnt/skills" literal in UserScopedSkillStorage.__init__ triggers test_skill_container_path_defaults::test_mnt_skills_literal_is_owned_by_skill_constants_module on CI. Migrate the default to the existing deerflow.constants constant, matching the pattern already used by LocalSkillStorage, SkillStorage, and the durable/tool_error middlewares. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Ryker_Feng
|
9124f991de
|
fix(mcp): make stdio MCP-produced files resolvable via virtual sandbox paths (#3597) (#3600)
* fix(mcp): migrate local MCP-produced files into sandbox outputs (#3597) Stdio MCP servers (e.g. Playwright) write files to host paths that the sandbox/artifact API cannot resolve, since it only serves paths under /mnt/user-data. Copy local files referenced by ResourceLink results into the thread's sandbox outputs dir and rewrite their URIs to /mnt/user-data/outputs/... so they become readable. Also scope pooled MCP sessions by user_id:thread_id instead of thread_id alone, matching the per-(user_id, thread_id) filesystem isolation. * fix(mcp): restrict file migration to trusted source roots (#3597) Add a source-root allowlist to the MCP file-migration path so a malicious or buggy MCP server cannot have us copy arbitrary host files (e.g. /etc/passwd) into a thread's outputs directory, from where the artifact API would serve them. Files are migrated only when located under the OS temp dir (Playwright's default), the thread's own user-data tree, or an operator-configured root via DEERFLOW_MCP_MIGRATION_SOURCE_ROOTS. Expand test coverage with allowlist/security cases (path escape refusal, trusted-root acceptance), URL-encoded file:// paths, converter content branches (image/embedded/error/structured), and copy/resolve failure fallbacks. * fix(mcp): harden local file migration into sandbox outputs Address robustness and security gaps in the MCP ResourceLink file migration: - Set migrated files to 0o644 so a differently-UID sandbox container can read them, instead of inheriting the source's (possibly 0o600) mode. - Enforce the 100MB size cap during the copy (chunked, byte-counted) rather than from a prior stat(), closing the grow-after-stat TOCTOU. - Create the destination atomically with O_CREAT|O_EXCL to remove the check-then-create name-collision race. - Document the shared-$TMPDIR multi-tenant read surface and mitigation. Add regression tests: symlink escape refusal, explicit $TMPDIR source migration, 0o644 mode, and the outputs/user-data resolve() OSError fallback branches. * fix(mcp): migrate playwright text file outputs * fix(mcp): translate MCP file outputs to virtual paths instead of copying (#3597) Pin stdio MCP subprocess cwd and TMPDIR/TMP/TEMP under the thread workspace so produced files always land in the mounted user-data tree, then rewrite returned references via deterministic host->virtual path translation. Free text is best-effort only: a reference is rewritten only when it resolves to an existing file inside the thread's tree, and bare filenames are matched against files created/modified by the same tool call. Replaces the previous copy-into-outputs + regex approach (which missed cases like temp/page-*.yml). * style(mcp): apply ruff format to mcp path translation tests * perf(mcp): offload stdio FS work off event loop and gate on transport Address review on #3600: - Wrap the workspace dir prep, snapshot diff, and per-token path resolution in asyncio.to_thread so they no longer block the event loop (matches the repo's blocking-IO gate convention). - Gate the cwd/temp pinning and snapshots on stdio transport only; SSE/HTTP servers skip the filesystem work entirely. - Skip the post-call snapshot diff when the result has no text content. * test(mcp): cover stdio transport gating and text-content after-walk skip Add unit/integration coverage for the new review-driven behavior: - _prepare_stdio_workspace dir/temp/snapshot bundle - _result_has_text_content detection (text, embedded text, image, empty) - non-stdio transport skips cwd/temp pinning and touches no workspace dirs - post-call snapshot diff is skipped without text content and runs with it * fix(mcp): address stdio path rewrite review feedback - Restrict the stdio MCP temp directory to 0700 instead of 0777. - Preserve operator-provided stdio cwd values while keeping injected cwd values as strings. - Add debug logging for deterministic path rewrites and bare-filename rewrite decisions. - Document the stdio cwd/temp pinning, virtual-path translation, and user/thread session scope. - Cover explicit cwd preservation and temp-dir permissions in session-pool tests. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |