mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-23 13:06:20 +00:00
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
betterkite
|
b552b5015c
|
fix(messages): drop legacy <uploaded_files> tag handling (#4826)
* fix(messages): drop legacy <uploaded_files> tag handling (#4212) PR #4174 unified upload-context injection on <current_uploads> (IM and web both flow through UploadsMiddleware), and #4632 documented the current path. This removes the remaining backward-compat parsing of the pre-#4174 <uploaded_files> tag, the final cleanup item tracked by the issue: - deermem: only <current_uploads> is stripped from human turns before memory persistence, and the upload-sentence scrubber drops the legacy tag alternative. - mem0: the mirrored message filter recognises only <current_uploads>. - InputSanitizationMiddleware: remove the legacy tag from the blocked-tag denylist (it existed only because deermem parsed the old tag). - frontend: stripUploadedFilesTag / stripInternalMarkers / parseUploadedFiles and the message-list fallback parse only <current_uploads>; demo thread fixtures are migrated to the current tag. Scope decision: a <uploaded_files> block in pre-#4174 history is now treated as ordinary user content (pinned by tests in both layers) instead of being silently dropped or stripped. * style: apply prettier formatting to stripUploadedFilesTag * fix(uploads): keep legacy <uploaded_files> stripping for display/export only Addresses review feedback on #4826: removing the legacy tag from the frontend display layer made pre-#4174 threads render raw <uploaded_files> XML (with server-side upload paths) in chat, copy data, and JSON exports. The backend cleanup stands — memory pipelines and the sanitization denylist treat only <current_uploads> as an internal marker. The frontend keeps the legacy spelling in its display/export-only utilities (stripUploadedFilesTag / INTERNAL_MARKER_TAGS / parseUploadedFiles and the message-list fallback) so old history renders cleanly without leaking internal paths, while the memory/sanitization scope-decision tests remain unchanged. Frontend tests now pin both spellings: <current_uploads> and legacy <uploaded_files> are stripped from copy data, markdown leak-stripping, and JSON exports. * docs(ui): record accepted display-spoof tradeoff for legacy upload tag Review note (willem-bd): since <uploaded_files> is off the sanitization denylist, a live user can type the legacy spelling and fabricate file chips / hide their own message text in display. Display-only and self-inflicted with no backend semantics, so it is accepted for now; documented at both the message-list fallback and stripUploadedFilesTag. Age-gating the legacy spelling remains a possible follow-up. --------- Co-authored-by: betterkite <313258397+betterkite@users.noreply.github.com> |
||
|
georgelichen
|
641a4147e7
|
fix(skills): parse Responses API content blocks in moderation scanner (#4936)
* Fix skill moderation parsing for Responses API content blocks Normalize LangChain Responses API text blocks before parsing the security moderation decision, while preserving the existing fail-closed behavior for unavailable or invalid moderation results. Add regression coverage for mixed content blocks and document the compatibility boundary. Constraint: Responses API AIMessage content is list-shaped while Chat Completions content is string-shaped Rejected: Disable security scanning | would weaken the skill write safety boundary Confidence: high Scope-risk: narrow Reversibility: clean Directive: Keep moderation parsing provider-format tolerant without including reasoning or tool blocks in the decision payload Tested: 27 security scanner tests; ruff check; ruff format check Not-tested: Live moderation request against the configured external endpoint * Reuse shared LLM response text normalization Route skill moderation responses through the existing provider-format normalizer so only text and output_text blocks participate in JSON parsing. Strengthen regression coverage with reasoning and tool blocks that contain misleading text fields.\n\nConstraint: Responses API content is shared across multiple harness consumers\nRejected: Keep a private normalizer | duplicated provider-shape policy diverges and can reintroduce reasoning-block contamination\nConfidence: high\nScope-risk: narrow\nReversibility: clean\nDirective: Extend the shared normalizer when a new provider content shape is verified; do not add divergent local parsers\nTested: 118 related backend tests; regression test red against the previous parser; Ruff check and format check\nNot-tested: Live GitHub CLA status refresh * Restore trusted external skill package loading Skill discovery follows one-level package-directory symlinks, but activation path validation rejected the resolved external path. Restore that compatibility for configured custom-skill category roots while keeping file-level symlinks and deeper escapes blocked. Add regression coverage for local and user-scoped storage plus slash activation, and document the boundary. Constraint: Existing skill discovery follows directory symlinks and operator-managed external packages must remain loadable Rejected: Allow arbitrary resolved paths | would weaken the skill path trust boundary Confidence: high Scope-risk: moderate Directive: Keep the final SKILL.md file symlink-free and preserve one-level category-root validation Tested: 79 targeted skill storage, loader, slash activation, and user-scoped tests passed; Ruff check and format check passed; GitNexus staged change detection reported low risk Not-tested: Real symlink activation on this Windows host lacks SeCreateSymbolicLinkPrivilege and is skipped Related: Skill projection copies sources into sandbox-visible views * Exercise real filesystem symlink boundaries in skill storage tests Replace global Path.resolve/is_symlink mocks with real directory and file symlinks, preserving the Windows privilege skip. Add regression coverage for deeper custom-root escapes and symlinks under non-custom categories so the one-level allowance remains explicit. Constraint: Symlink creation requires SeCreateSymbolicLinkPrivilege on some Windows runners Rejected: Keep global path-method mocks | they validate the mock behavior rather than filesystem semantics Confidence: high Scope-risk: narrow Reversibility: clean Directive: Keep security-boundary tests on real filesystem primitives; skip only when the runner lacks symlink privilege Tested: 76 targeted loader/storage/slash tests; Ruff check; Ruff format check Not-tested: Windows symlink-enabled execution on this host Related: #4936 * Pin the actual nested symlink escape boundary Place the second symlink below a real custom package directory so the test reaches the one-level relative-parent guard instead of returning early on a non-symlink parent. Keep the public-category rejection coverage unchanged. Constraint: The security boundary depends on both symlink depth and category root Rejected: Link the outer package directory directly | the parent is not a symlink at validation time, so the depth guard is never evaluated Confidence: high Scope-risk: narrow Reversibility: clean Directive: Keep this regression tied to the exact relative_parent.parts depth check Tested: Targeted storage, loader, and slash suites; GitNexus staged detection Not-tested: Symlink-enabled execution on this Windows host Related: #4936 * Make the nested symlink regression reach the depth guard The test now validates the SKILL.md directly through the nested symlink, so the symlink is the immediate parent and the relative-parent depth check is executed. Constraint: Windows test execution may skip when symlink privilege is unavailable Rejected: Keep the extra nested path segment | it bypasses the symlink-depth guard through an early return Confidence: high Scope-risk: narrow Reversibility: clean Directive: Mutation tests must fail when the depth restriction is removed Tested: Targeted test (skipped on this Windows host without symlink privilege); Ruff check and format check Not-tested: Real symlink execution on Windows; Linux CI will exercise the case Related: #4936 * Keep sandbox projections fresh for linked external skill packages The storage layer intentionally accepts one-level custom package-directory symlinks, but projection freshness previously hashed only the link inode. Follow the permitted target tree during custom and legacy source-signature scans so edits to SKILL.md, scripts, references, or assets trigger a rebuild before sandbox use. Constraint: Preserve the existing one-level custom/legacy symlink boundary and do not follow public, integration, nested, or file symlinks Rejected: Invalidate projections only from /api/skills/reload | sandbox acquisition must also detect edits made directly in external targets Confidence: high Scope-risk: narrow Reversibility: clean Directive: Keep target-tree traversal limited to the storage paths that explicitly permit external package-directory links Tested: 77 projection, user-scoped storage, and lifecycle tests passed; Ruff check and format check passed; git diff --check passed; GitNexus staged detection reported low risk Not-tested: Real external symlink execution on this Windows host without SeCreateSymbolicLinkPrivilege; existing tests skip that platform limitation |
||
|
Huixin615
|
d726ae60c3
|
fix(skills): use managed integrations root for slash activation (#4570)
* fix(skills): use managed integrations root for slash activation * refactor(skills): clarify integrations root getter |
||
|
MiaoRuidx
|
f1632cc351
|
fix(run): add run event stream contract (#4342)
* docs: document run event stream contract * fix(run): address event stream review feedback --------- Co-authored-by: MiaoRuidx <12540796+MiaoRuidx@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Huixin615
|
65afc9b1d2
|
fix(skills): apply allowed-tools only to active skills (#4098)
* fix(skills): scope allowed-tools to active skills * fix(skills): tolerate stale active skill paths * chore: retrigger CI * fix(skills): document policy activation limits * perf(skills): reuse per-step tool policy decisions * fix(skills): harden runtime tool policy contracts * fix(skills): redact cached policy decisions * fix(skills): make slash tool policy authoritative * fix(skills): preserve policy-safe discovery tools * test(skills): cover explicit task delegation policy |
||
|
Daoyuan Li
|
2fa0505070
|
fix(skills): activate a slash skill once per run, not per model call (#4103)
* fix(skills): activate a slash skill once per run, not per model call SkillActivationMiddleware injects the activation reminder for a slash command via request.override(messages=...), which LangChain's create_agent uses for a single model call and never writes back to graph state. The dedup guard scans request.messages for a prior reminder, but model_node rebuilds request.messages fresh from persisted state on every tool-loop step, so the reminder is never present on the 2nd..Nth model call of a turn. Every model call therefore re-parsed the command, re-read SKILL.md from disk, re-injected the multi-KB body, and re-recorded an "activate" audit event, despite the code intending a single activation per run (#3861 semantics: one activation call, many follow-up model calls). Key the dedup off the run context instead, which LangGraph threads through every model-node call of a run (the same durable signal the request-scoped secret source already uses). The activation call records the slash message's identity in context; later calls for the same message skip re-activation. A new user slash message keys differently and still activates. Secret binding is unaffected: it already re-resolves from the persisted slash source on every call. Adds regression tests that rebuild the real multi-call turn state and assert a single activation across the tool loop, plus a test proving a new slash command still activates. * fix(skills): address review nits on run-scoped activation dedup - Extract _already_activated(run_context, run_key) so the dedup check mirrors the existing _has_existing_activation_for_target sibling instead of an inline dense conditional. - Compute _activation_run_key() once in _find_activation_target and thread it through _prepare_model_request instead of recomputing it at the write site, making the "same key for check and write" invariant explicit in the code rather than implicit. - Document why the run-context write is an overwrite rather than an append/set: only the latest real user message is ever considered an activation target, so there is nothing earlier in the run worth preserving. - Add a regression test locking in the degraded-path contract: when runtime.context is None, the middleware still activates per-call instead of crashing or wrongly no-op'ing. |
||
|
Aari
|
42544755ac
|
fix(skills): escape untrusted skill metadata before it enters the model prompt (#4128)
* fix(skills): escape untrusted skill metadata before it enters the model prompt Skill name/description/allowed-tools come from the frontmatter of a user-installable .skill archive (POST /api/skills/install or a drop into skills/custom/); the parser only strips them. The slash-activation and durable-context siblings already html.escape these exact fields before rendering them into a model-visible block -- but five other render sites emit them raw. The sharpest is the default path, <available_skills> in the system prompt (skills.deferred_discovery: false): a community skill whose description closes the block can forge a framework-trusted <system-reminder> into the lead-agent system prompt. Driven through the real apply_prompt_template(), the forged tag reaches the system prompt raw on main and is neutralized here. Escape at every render site that emits untrusted skill metadata/content: - <available_skills> (name/description/location) and <disabled_skills> (name) in lead_agent/prompt.py; - describe_skill output (name/description/allowed-tools/location) and <skill_index> (name) in skills/describe.py; - the subagent <skill name=...> attribute plus the raw SKILL.md body in subagents/executor.py::_load_skill_messages -- its direct sibling skill_activation escapes both, this escaped neither. quote=False in element-text positions (matching skill_context and the #4097 correction), quote=True in the one attribute position (matching skill_activation). category is a controlled enum and is left as-is; escaping is render-time only, so stored skills are unchanged and re-rendering never double-escapes. * fix(skills): escape skill name in the slash-activation prose line The slash-activation reminder emitted `activation.skill_name` raw in its prose line while escaping the same value in the adjacent <skill name="..."> attribute. skill_name is grammar-gated to [a-z0-9-] by resolve_slash_skill before it reaches the renderer, so this is a defense-in-depth / consistency fix rather than a reachable injection: the two positions can never drift if a future caller builds an activation from an unconstrained name. Reuse the already-computed escaped_skill_name. |
||
|
Tianye Song
|
15454b6fec
|
feat(skills): deferred skill discovery via describe_skill tool (#3775)
Replace the full-metadata <available_skills> system-prompt block with a compact <skill_index> (names only) and an on-demand describe_skill tool when skills.deferred_discovery: true (default: false / backward compat). New modules: - skills/catalog.py — SkillCatalog (immutable, searchable; select: has no cap, keyword/prefix search caps at MAX_RESULTS=5) - skills/describe.py — build_describe_skill_tool(catalog) closure; build_skill_search_setup() wires SkillSearchSetup into both the LangGraph agent factory (agent.py) and DeerFlowClient (client.py) Changes: - Skill @dataclass(frozen=True); allowed_tools/required_secrets list→tuple - Skill First prompt line gated on skill_names (deferred vs legacy wording) - get_skills_prompt_section: short-circuit storage on deferred path; merge user_id (upstream) + skill_names (this PR) params - describe_skill tool parameter named "name" (matches prompt wording) - select: branch removes [:MAX_RESULTS] cap (exact request, not ranking) - AGENTS.md: document deferred_discovery config field + new modules Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zhipeng Zheng
|
53a80d3ad1
|
feat(skills): per-user custom skill isolation with sandbox mounting (#3889)
* feat(skills): per-user skill isolation (#2905) Implement user-scoped skill storage that isolates custom skills between users while sharing public skills globally. Key changes: - Add UserScopedSkillStorage class for per-user custom skill directories - Introduce get_or_new_user_skill_storage() factory with user_id context - Auth middleware sets effective_user_id for request-scoped storage - Agent/prompt/middleware now use user-scoped storage and prompt cache - Sandbox mounts user-scoped skill directories for search/read tools - Add validate_skill_file_path() to SkillStorage for path security - Migration script supports --all-users bulk migration - Frontend: add editable field to Skill type, error check in enableSkill - All skill categories can be toggled (custom skills default to enabled) - Update skill-creator SKILL.md with isolation-aware instructions Tests: - Add test_user_scoped_skill_storage.py (new) - Update all existing skill tests for user-scoped storage - Update sandbox, client, and router tests * fix(skills): address second-round PR review feedback (#3889) - P1-1: restrict legacy skill mount to users without custom skills - P1-2: fail-closed for _is_disabled_skill_path (OSError → return True) - P2-1: AND-merge global extensions_config skill disabled state - P2-2: atomic write for _skill_states.json (mkstemp + replace) - P2-3: normalize X-DeerFlow-Owner-User-Id in trusted boundary - P2-4: LRU-bounded _enabled_skills_by_config_cache (OrderedDict, maxsize=256) - P2-5: clear global prompt cache on PUBLIC skill toggle - P2-6: invalidate skill caches on client.update_skill * fix(tests): correct tool policy test after merge * fix(skills): use DEFAULT_SKILLS_CONTAINER_PATH in UserScopedSkillStorage The "/mnt/skills" literal in UserScopedSkillStorage.__init__ triggers test_skill_container_path_defaults::test_mnt_skills_literal_is_owned_by_skill_constants_module on CI. Migrate the default to the existing deerflow.constants constant, matching the pattern already used by LocalSkillStorage, SkillStorage, and the durable/tool_error middlewares. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
DanielWalnut
|
25ea6970a6
|
feat(runtime): implement goal continuations (#3858)
* implement goal continuations * fix(goal): address review findings for goal continuations - goal: key the no-progress breaker on a signature of the latest visible assistant evidence instead of the evaluator's volatile free-text, so it actually fires on stalled turns; thread the signature through every worker persist / no-progress call site - goal: align _stand_down_reason default caps with should_continue_goal (8 / 2) so the two gate functions agree on goals missing the fields - runtime: offload the synchronous checkpointer fallback via asyncio.to_thread (goal.py + worker.py) to keep blocking IO off the loop - frontend: i18n the GoalStatus "Goal" label (goalLabel in en/zh/types) - frontend: extract pure composer helpers into input-box-helpers.ts with unit tests (parseGoalCommand, readGoalResponseError, skill suggestions) - tests: cover the evidence-based no-progress and default-cap behavior - docs: align backend/AGENTS.md goal paragraph with actual behavior - e2e: prettier-format chat.spec.ts (fixes the lint-frontend CI failure) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): hide goal continuation counter until the agent continues The goal status bar rendered a raw "0/8" before any auto-continuation, which read as a mysterious score. Now the counter is hidden until continuation_count > 0, then shows "Continuing N/M" with a tooltip explaining the auto-continuation cap. - Extract getGoalContinuationDisplay into a pure helper (hides at 0) + unit tests - Add goalContinuing / goalContinuationTooltip i18n keys (en/zh/types) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(goal): address review findings for goal continuations Frontend correctness - Fix the optimistic /goal result permanently shadowing server goal state: the streamed continuation counter never surfaced for a goal set in-session. Extract a shared useActiveGoal hook (used by both chat pages) that reconciles the optimistic copy with server state via a goalReconciliationKey, de-duping the copy-pasted goal block across the two pages. - Stop /goal status|clear failures from escaping handleSubmit as unhandled rejections (handleGoalCommand now returns success; the run only starts when a goal was actually saved). - Use a function replacer for the goal-status toast so an objective containing $&/$1 isn't treated as a replacement pattern. Backend cleanliness / correctness - De-duplicate four byte-identical helpers (_call_checkpointer_method, _message_type, _additional_kwargs, _is_visible_message) by importing them from runtime.goal instead of re-defining them in the run worker. - Remove the dead `checkpoint_tuple.tasks` durability guard (CheckpointTuple has no tasks field) and document that pending_writes is the durability signal. - Decompose the 176-line _prepare_goal_continuation_input: extract _reread_goal_and_checkpoint and a _persist closure so the thread-unchanged guard and stand-down persistence aren't open-coded three times. Document the last-writer-wins write-window limitation as a follow-up. - Add a shared parse_goal_command helper and use it from the TUI and IM-channel /goal handlers (one place for the status/clear/set semantics). Tests - Restore the 11 command-registry tests dropped by the previous goal change (filter_commands ranking/description, build_registry builtins/skills, resolve cases) alongside the new goal tests. - Add coverage for the IM-channel _handle_goal_command, the TUI _handle_goal handler, parse_goal_command, and goalReconciliationKey. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix goal review feedback * fix goal continuation checkpoint races * prioritize goal commands while streaming Route composer submits through a shared helper so /goal commands can be handled before the streaming stop shortcut, while ordinary streaming submits still stop the active run. Testing: cd frontend && pnpm exec rstest run tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; cd frontend && pnpm check * preserve goal status during clarification Keep omitted stream goal fields distinct from explicit null clears so clarification interrupts do not hide an active thread goal that is still present in the checkpoint. Testing: pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check * style: format active goal hook Run Prettier on use-active-goal.ts to satisfy the frontend lint workflow formatting gate. Testing: pnpm format; pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check * fix goal review followups --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
AochenShen99
|
442248dd06
|
feat: preserve durable context across summarization (#3887)
* feat: preserve durable context across summarization * fix: harden durable context review gaps * style: format delegation ledger live test * chore: remove stale delegation ledger prefix * fix: address durable context review feedback |
||
|
DanielWalnut
|
16391e35ab
|
fix(skills): harden slash skill activation across chat channels (#3466)
* support slash skill activation * format slash skill activation * Preserve slash skill activation with uploads * Address slash skill review feedback * Address slash skill follow-up review * Fix lazy slash skill storage resolution * Keep slash skill activation out of system prompt * Address slash skill review issues * fix: harden slash skill command handling * feat(frontend): add slash skill autocomplete * fix: address slash skill review feedback * fix: preserve slash skill text for IM uploads |