153 Commits

Author SHA1 Message Date
Async23
0948c7a4e1
fix(provider): preserve streamed Codex output when response.completed.output is empty (#1928)
* fix: preserve streamed Codex output items

* fix: prefer completed Codex output over streamed placeholders
2026-04-07 18:21:22 +08:00
koppx
c3170f22da
fix(backend): make loop detection hash tool calls by stable keys (#1911)
* fix(backend): make loop detection hash tool calls by stable keys

The loop detection middleware previously hashed full tool call arguments,
which made repeated calls look different when only non-essential argument
details changed. In particular, `read_file` calls with nearby line ranges
could bypass repetition detection even when the agent was effectively
reading the same file region again and again.

- Hash tool calls using stable keys instead of the full raw args payload
- Bucket `read_file` line ranges so nearby reads map to the same region key
- Prefer stable identifiers such as `path`, `url`, `query`, or `command`
  before falling back to JSON serialization of args
- Keep hashing order-independent so the same tool call set produces the
  same hash regardless of call order

Fixes #1905

* fix(backend): harden loop detection hash normalization

- Normalize and parse stringified tool args defensively
- Expand stable key derivation to include pattern, glob, and cmd
- Normalize reversed read_file ranges before bucketing

Fixes #1905

* fix(backend): harden loop detection tool format

* exclude write_file and str_replace from the stable-key path — writing different content to the same file shouldn't be flagged.

---------

Co-authored-by: JeffJiang <for-eleven@hotmail.com>
2026-04-07 17:46:33 +08:00
KKK
3b3e8e1b0b
feat(sandbox): strengthen bash command auditing with compound splitting and expanded patterns (#1881)
* fix(sandbox): strengthen regex coverage in SandboxAuditMiddleware

Expand high-risk patterns from 6 to 13 and medium-risk from 4 to 6,
closing several bypass vectors identified by cross-referencing Claude
Code's BashSecurity validator chain against DeerFlow's threat model.

High-risk additions:
- Generalised pipe-to-sh (replaces narrow curl|sh rule)
- Targeted command substitution ($() / backtick with dangerous executables)
- base64 decode piped to execution
- Overwrite system binaries (/usr/bin/, /bin/, /sbin/)
- Overwrite shell startup files (~/.bashrc, ~/.profile, etc.)
- /proc/*/environ leakage
- LD_PRELOAD / LD_LIBRARY_PATH hijack
- /dev/tcp/ bash built-in networking

Medium-risk additions:
- sudo/su (no-op under Docker root, warn only)
- PATH= modification (long attack chain, warn only)

Design decisions:
- Command substitution uses targeted matching (curl/wget/bash/sh/python/
  ruby/perl/base64) rather than blanket block to avoid false positives
  on safe usage like $(date) or `whoami`.
- Skipped encoding/obfuscation checks (hex, octal, Unicode homoglyphs)
  as ROI is low in Docker sandbox — LLMs don't generate encoded commands
  and container isolation bounds the blast radius.
- Merged pip/pip3 into single pip3? pattern.

* feat(sandbox): compound command splitting and fork bomb detection

Split compound bash commands (&&, ||, ;) into sub-commands and classify
each independently — prevents dangerous commands hidden after safe
prefixes (e.g. "cd /workspace && rm -rf /") from bypassing detection.

- Add _split_compound_command() with shlex quote-aware splitting
- Add fork bomb detection patterns (classic and while-loop variants)
- Most severe verdict wins; block short-circuits
- 15 new tests covering compound commands, splitting, and fork bombs

* test(sandbox): add async tests for fork bomb and compound commands

Cover awrap_tool_call path for fork bomb detection (3 variants) and
compound command splitting (block/warn/pass scenarios).

* fix(sandbox): address Copilot review — no-whitespace operators, >>/etc/, whole-command scan

- _split_compound_command: replace shlex-based implementation with a
  character-by-character quote/escape-aware scanner. shlex.split only
  separates '&&' / '||' / ';' when they are surrounded by whitespace,
  so payloads like 'rm -rf /&&echo ok' or 'safe;rm -rf /' bypassed the
  previous splitter and therefore the per-sub-command classifier.
- _HIGH_RISK_PATTERNS: change r'>\s*/etc/' to r'>+\s*/etc/' so append
  redirection ('>>/etc/hosts') is also blocked.
- _classify_command: run a whole-command high-risk scan *before*
  splitting. Structural attacks like 'while true; do bash & done'
  span multiple shell statements — splitting on ';' destroys the
  pattern context, so the raw command must be scanned first.
- tests: add no-whitespace operator cases to TestSplitCompoundCommand
  and test_compound_command_classification to lock in the bypass fix.
2026-04-07 17:15:24 +08:00
lulusiyuyu
f0dd8cb0d2
fix(subagents): add cooperative cancellation for subagent threads (#1873)
* fix(subagents): add cooperative cancellation for subagent threads

Subagent tasks run inside ThreadPoolExecutor threads with their own
event loop (asyncio.run). When a user clicks stop, RunManager cancels
the parent asyncio.Task, but Future.cancel() cannot terminate a running
thread and asyncio.Event does not propagate across event loops. This
causes subagent threads to keep executing (writing files, calling LLMs)
even after the user explicitly stops the run.

Fix: add a threading.Event (cancel_event) to SubagentResult and check
it cooperatively in _aexecute()'s astream iteration loop. On cancel,
request_cancel_background_task() sets the event, and the thread exits
at the next iteration boundary.

Changes:
- executor.py: Add cancel_event field to SubagentResult, check it in
  _aexecute loop, set it on timeout, add request_cancel_background_task
- task_tool.py: Call request_cancel_background_task on CancelledError

* fix(subagents): guard cancel status and add pre-check before astream

- Only overwrite status to FAILED when still RUNNING, preserving
  TIMED_OUT set by the scheduler thread.
- Add cancel_event pre-check before entering the astream loop so
  cancellation is detected immediately when already signalled.

* fix(subagents): guard status updates with lock to prevent race condition

Wrap the check-and-set on result.status in _aexecute with
_background_tasks_lock so the timeout handler in execute_async
cannot interleave between the read and write.

* fix(subagents): add dedicated CANCELLED status for user cancellation

Introduce SubagentStatus.CANCELLED to distinguish user-initiated
cancellation from actual execution failures.  Update _aexecute,
task_tool polling, cleanup terminal-status sets, and test fixtures.

* test(subagents): add cancellation tests and fix timeout regression test

- Add dedicated TestCooperativeCancellation test class with 6 tests:
  - Pre-set cancel_event prevents astream from starting
  - Mid-stream cancel_event returns CANCELLED immediately
  - request_cancel_background_task() sets cancel_event correctly
  - request_cancel on nonexistent task is a no-op
  - Real execute_async timeout does not overwrite CANCELLED (deterministic
    threading.Event sync, no wall-clock sleeps)
  - cleanup_background_task removes CANCELLED tasks

- Add task_tool cancellation coverage:
  - test_cancellation_calls_request_cancel: assert CancelledError path
    calls request_cancel_background_task(task_id)
  - test_task_tool_returns_cancelled_message: assert CANCELLED polling
    branch emits task_cancelled event and returns expected message

- Fix pre-existing test infrastructure issue: add deerflow.sandbox.security
  to _MOCKED_MODULE_NAMES (fixes ModuleNotFoundError for all executor tests)

- Add RUNNING guard to timeout handler in executor.py to prevent
  TIMED_OUT from overwriting CANCELLED status

- Add cooperative cancellation granularity comment documenting that
  cancellation is only detected at astream iteration boundaries

---------

Co-authored-by: lulusiyuyu <lulusiyuyu@users.noreply.github.com>
2026-04-07 11:12:25 +08:00
DanielWalnut
7643a46fca
fix(skill): make skill prompt cache refresh nonblocking (#1924)
* fix: make skill prompt cache refresh nonblocking

* fix: harden skills prompt cache refresh

* chore: add timeout to skills cache warm-up
2026-04-07 10:50:34 +08:00
Markus Corazzione
c4da0e8ca9
Move async SQLite mkdir off the event loop (#1921)
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
2026-04-07 10:47:20 +08:00
JilongSun
88e535269e
Feature/feishu receive file (#1608)
* feat(feishu): add channel file materialization hook for inbound messages

- Introduce Channel.receive_file(msg, thread_id) as a base method for file materialization; default is no-op.
- Implement FeishuChannel.receive_file to download files/images from Feishu messages, save to sandbox, and inject virtual paths into msg.text.
- Update ChannelManager to call receive_file for any channel if msg.files is present, enabling downstream model access to user-uploaded files.
- No impact on Slack/Telegram or other channels (they inherit the default no-op).

* style(backend): format code with ruff for lint compliance

- Auto-formatted packages/harness/deerflow/agents/factory.py and tests/test_create_deerflow_agent.py using `ruff format`
- Ensured both files conform to project linting standards
- Fixes CI lint check failures caused by code style issues

* fix(feishu): handle file write operation asynchronously to prevent blocking

* fix(feishu): rename GetMessageResourceRequest to _GetMessageResourceRequest and remove redundant code

* test(feishu): add tests for receive_file method and placeholder replacement

* fix(manager): remove unnecessary type casting for channel retrieval

* fix(feishu): update logging messages to reflect resource handling instead of image

* fix(feishu): sanitize filename by replacing invalid characters in file uploads

* fix(feishu): improve filename sanitization and reorder image key handling in message processing

* fix(feishu): add thread lock to prevent filename conflicts during file downloads

* fix(test): correct bad merge in test_feishu_parser.py

* chore: run ruff and apply formatting cleanup
fix(feishu): preserve rich-text attachment order and improve fallback filename handling
2026-04-06 22:14:12 +08:00
DanielWalnut
888f7bfb9d
Implement skill self-evolution and skill_manage flow (#1874)
* chore: ignore .worktrees directory

* Add skill_manage self-evolution flow

* Fix CI regressions for skill_manage

* Address PR review feedback for skill evolution

* fix(skill-evolution): preserve history on delete

* fix(skill-evolution): tighten scanner fallbacks

* docs: add skill_manage e2e evidence screenshot

* fix(skill-manage): avoid blocking fs ops in session runtime

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-06 22:07:11 +08:00
KKK
055e4df049
fix(sandbox): add input sanitisation guard to SandboxAuditMiddleware (#1872)
* fix(sandbox): add L2 input sanitisation to SandboxAuditMiddleware

Add _validate_input() to reject malformed bash commands before regex
classification: empty commands, oversized commands (>10 000 chars), and
null bytes that could cause detection/execution layer inconsistency.

* fix(sandbox): address Copilot review — type guard, log truncation, reject reason

- Coerce None/non-string command to str before validation
- Truncate oversized commands in audit logs to prevent log amplification
- Propagate reject_reason through _pre_process() to block message
- Remove L2 label from comments and test class names

* fix(sandbox): isinstance type guard + async input sanitisation tests

Address review comments:
- Replace str() coercion with isinstance(raw_command, str) guard so
  non-string truthy values (0, [], False) fall back to empty string
  instead of passing validation as "0"/"[]"/"False".
- Add TestInputSanitisationBlocksInAwrapToolCall with 4 async tests
  covering empty, null-byte, oversized, and None command via
  awrap_tool_call path.
2026-04-06 17:21:58 +08:00
Zhou
1ced6e977c
fix(backend): preserve viewed image reducer metadata (#1900)
Fix concurrent viewed_images state updates for multi-image input by preserving the reducer metadata in the vision middleware state schema.
2026-04-06 16:47:19 +08:00
NmanQAQ
dd30e609f7
feat(models): add vLLM provider support (#1860)
support for vLLM 0.19.0 OpenAI-compatible chat endpoints and fixes the Qwen reasoning toggle so flash mode can actually disable thinking.

Co-authored-by: NmanQAQ <normangyao@qq.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-06 15:18:34 +08:00
yangzheli
5fd2c581f6
fix: add output truncation to ls_tool to prevent context window overflow (#1896)
ls_tool was the only sandbox tool without output size limits, allowing
multi-MB results from large directories to blow up the model context
window. Add head-truncation (configurable via ls_output_max_chars,
default 20000) consistent with existing bash and read_file truncation.

Closes #1887

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-06 15:09:57 +08:00
7c68dd4ad4
Fix(#1702): stream resume run (#1858)
* fix: repair stream resume run metadata

# Conflicts:
#	backend/packages/harness/deerflow/runtime/stream_bridge/memory.py
#	frontend/src/core/threads/hooks.ts

* fix(stream): repair resumable replay validation

---------

Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-06 14:51:10 +08:00
suyua9
29575c32f9
fix: expose custom events from DeerFlowClient.stream() (#1827)
* fix: expose custom client stream events

Signed-off-by: suyua9 <1521777066@qq.com>

* fix(client): normalize streamed custom mode values

* test(client): satisfy backend ruff import ordering

---------

Signed-off-by: suyua9 <1521777066@qq.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-06 10:09:39 +08:00
Chris Z
117fa9b05d
fix(channels): normalize slack allowed user ids (#1802)
* fix(channels): normalize slack allowed user ids

* style(channels): apply backend formatter

---------

Co-authored-by: haimingZZ <15558128926@qq.com>
Co-authored-by: suyua9 <1521777066@qq.com>
2026-04-05 18:04:21 +08:00
thefoolgy
8049785de6
fix(memory): case-insensitive fact deduplication and positive reinforcement detection (#1804)
* fix(memory): case-insensitive fact deduplication and positive reinforcement detection

Two fixes to the memory system:

1. _fact_content_key() now lowercases content before comparison, preventing
   semantically duplicate facts like "User prefers Python" and "user prefers
   python" from being stored separately.

2. Adds detect_reinforcement() to MemoryMiddleware (closes #1719), mirroring
   detect_correction(). When users signal approval ("yes exactly", "perfect",
   "完全正确", etc.), the memory updater now receives reinforcement_detected=True
   and injects a hint prompting the LLM to record confirmed preferences and
   behaviors with high confidence.

   Changes across the full signal path:
   - memory_middleware.py: _REINFORCEMENT_PATTERNS + detect_reinforcement()
   - queue.py: reinforcement_detected field in ConversationContext and add()
   - updater.py: reinforcement_detected param in update_memory() and
     update_memory_from_conversation(); builds reinforcement_hint alongside
     the existing correction_hint

Tests: 11 new tests covering deduplication, hint injection, and signal
detection (Chinese + English patterns, window boundary, conflict with correction).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(memory): address Copilot review comments on reinforcement detection

- Tighten _REINFORCEMENT_PATTERNS: remove 很好, require punctuation/end-of-string boundaries on remaining patterns, split this-is-good into stricter variants
- Suppress reinforcement_detected when correction_detected is true to avoid mixed-signal noise
- Use casefold() instead of lower() for Unicode-aware fact deduplication
- Add missing test coverage for reinforcement_detected OR merge and forwarding in queue

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-05 16:23:00 +08:00
Evan Wu
9ca68ffaaa
fix: preserve virtual path separator style (#1828)
* fix: preserve virtual path separator style

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-05 15:52:22 +08:00
Markus Corazzione
0ffe5a73c1
chroe(config):Increase subagent max-turn limits (#1852) 2026-04-05 15:41:00 +08:00
Octopus
a283d4a02d
fix: include soul field in GET /api/agents list response (fixes #1819) (#1863)
Previously, the list endpoint always returned soul=null because
_agent_config_to_response() was called without include_soul=True.
This caused confusion since PUT /api/agents/{name} and GET /api/agents/{name}
both returned the soul content, but the list endpoint silently omitted it.

Co-authored-by: octo-patch <octo-patch@users.noreply.github.com>
2026-04-05 10:49:58 +08:00
DanielWalnut
2a150f5d4a
fix: unblock concurrent threads and workspace hydration (#1839)
* fix: unblock concurrent threads and workspace hydration

* fix: restore async title generation

* fix: address PR review feedback

* style: format lead agent prompt
2026-04-04 21:19:35 +08:00
SHIYAO ZHANG
163121d327
fix(uploads): handle split-bold headings and ** ** artefacts in extract_outline (#1838)
* feat(uploads): guide agent to use grep/glob/read_file for uploaded documents

Add workflow guidance to the <uploaded_files> context block so the agent
knows to use grep and glob (added in #1784) alongside read_file when
working with uploaded documents, rather than falling back to web search.

This is the final piece of the three-PR PDF agentic search pipeline:
- PR1 (#1727): pymupdf4llm converter produces structured Markdown with headings
- PR2 (#1738): document outline injected into agent context with line numbers
- PR3 (this):  agent guided to use outline + grep + read_file workflow

* feat(uploads): add file-first priority and fallback guidance to uploaded_files context

* fix(uploads): handle split-bold headings and ** ** artefacts in extract_outline

- Add _clean_bold_title() to merge adjacent bold spans (** **) produced
  by pymupdf4llm when bold text crosses span boundaries
- Add _SPLIT_BOLD_HEADING_RE (Style 3) to recognise **<num>** **<title>**
  headings common in academic papers; excludes pure-number table headers
  and rows with more than 4 bold blocks
- When outline is empty, read first 5 non-empty lines of the .md as a
  content preview and surface a grep hint in the agent context
- Update _format_file_entry to render the preview + grep hint instead of
  silently omitting the outline section
- Add 3 new extract_outline tests and 2 new middleware tests (65 total)

* fix(uploads): address Copilot review comments on extract_outline regex

- Replace ASCII [A-Za-z] guard with negative lookahead to support non-ASCII
  titles (e.g. **1** **概述**); pure-numeric/punctuation blocks still excluded
- Replace .+ with [^*]+ and cap repetition at {0,2} (four blocks total) to
  keep _SPLIT_BOLD_HEADING_RE linear and avoid ReDoS on malformed input
- Remove now-redundant len(blocks) <= 4 code-level check (enforced by regex)
- Log debug message with exc_info when preview extraction fails
2026-04-04 14:25:08 +08:00
fengxsong
19809800f1
feat: support wecom channel (#1390)
* feat: support wecom channel

* fix: sending file to client

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* test: add unit tests for wecom channel

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* docs: add example configs and setup docs

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* revert pypi default index setting

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* revert: keeping codes in harness untouched

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* fix: format issue

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* fix: resolve Copilot comments

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

---------

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-04 11:28:35 +08:00
ppyt
db82b59254
fix(middleware): handle list-type AIMessage.content in LoopDetectionMiddleware (#1823)
* fix: inject longTermBackground into memory prompt

The format_memory_for_injection function only processed recentMonths and
earlierContext from the history section, silently dropping longTermBackground.

The LLM writes longTermBackground correctly and it persists to memory.json,
but it was never injected into the system prompt — making the user's
long-term background invisible to the AI.

Add the missing field handling and a regression test.

* fix(middleware): handle list-type AIMessage.content in LoopDetectionMiddleware

LangChain AIMessage.content can be str | list. When using providers that
return structured content blocks (e.g. Anthropic thinking mode, certain
OpenAI-compatible gateways), content is a list of dicts like
[{"type": "text", "text": "..."}].

The hard_limit branch in _apply() concatenated content with a string via
(last_msg.content or "") + f"\n\n{_HARD_STOP_MSG}", which raises
TypeError when content is a non-empty list (list + str is invalid).

Add _append_text() static method that:
- Returns the text directly when content is None
- Appends a {"type": "text"} block when content is a list
- Falls back to string concatenation when content is a str

This is consistent with how other modules in the project already handle
list content (client.py._extract_text, memory_middleware, executor.py).

* test(middleware): add unit tests for _append_text and list content hard stop

Add regression tests to verify LoopDetectionMiddleware handles list-type
AIMessage.content correctly during hard stop:

- TestAppendText: unit tests for the new _append_text() static method
  covering None, str, list (including empty list) content types
- TestHardStopWithListContent: integration tests verifying hard stop
  works correctly with list content (Anthropic thinking mode), None
  content, and str content

Requested by reviewer in PR #1823.

* fix(middleware): improve _append_text robustness and test isolation

- Add explicit isinstance(content, str) check with fallback for
  unexpected types (coerce to str) to prevent TypeError on edge cases
- Deep-copy list content in _make_state() test helper to prevent
  shared mutable references across test iterations
- Add test_unexpected_type_coerced_to_str: verify fallback for
  non-str/list/None content types
- Add test_list_content_not_mutated_in_place: verify _append_text
  does not modify the original list

* style: fix ruff format whitespace in test file

---------

Co-authored-by: ppyt <14163465+ppyt@users.noreply.github.com>
2026-04-04 10:38:22 +08:00
SHIYAO ZHANG
ddfc988bef
feat(uploads): add pymupdf4llm PDF converter with auto-fallback and async offload (#1727)
* feat(uploads): add pymupdf4llm PDF converter with auto-fallback and async offload

- Introduce pymupdf4llm as an optional PDF converter with better heading
  detection and table preservation than MarkItDown
- Auto mode: prefer pymupdf4llm when installed; fall back to MarkItDown
  when output is suspiciously sparse (image-based / scanned PDFs)
- Sparsity check uses chars-per-page (< 50 chars/page) rather than an
  absolute threshold, correctly handling both short and long documents
- Large files (> 1 MB) are offloaded to asyncio.to_thread() to avoid
  blocking the event loop (related: #1569)
- Add UploadsConfig with pdf_converter field (auto/pymupdf4llm/markitdown)
- Add pymupdf4llm as optional dependency: pip install deerflow-harness[pymupdf]
- Add 14 unit tests covering sparsity heuristic, routing logic, and async path

* fix(uploads): address Copilot review comments on PDF converter

- Fix docstring: MIN_CHARS_PYMUPDF -> _MIN_CHARS_PER_PAGE (typo)
- Fix file handle leak: wrap pymupdf.open in try/finally to ensure doc.close()
- Fix silent fallback gap: _convert_pdf_with_pymupdf4llm now catches all
  conversion exceptions (not just ImportError), so encrypted/corrupt PDFs
  fall back to MarkItDown instead of propagating
- Tighten type: pdf_converter field changed from str to Literal[auto|pymupdf4llm|markitdown]
- Normalize config value: _get_pdf_converter() strips and lowercases the raw
  config string, warns and falls back to 'auto' on unknown values
2026-04-03 21:59:45 +08:00
SHIYAO ZHANG
5ff230eafd
feat(uploads): inject document outline into agent context for converted files (#1738)
* feat(uploads): inject document outline into agent context for converted files

Extract headings from converted .md files and inject them into the
<uploaded_files> context block so the agent can navigate large documents
by line number before reading.

- Add `extract_outline()` to `file_conversion.py`: recognises standard
  Markdown headings (#/##/###) and SEC-style bold structural headings
  (**ITEM N. BUSINESS**, **PART II**); caps at 50 entries; excludes
  cover-page boilerplate (WASHINGTON DC, CURRENT REPORT, SIGNATURES)
- Add `_extract_outline_for_file()` helper in `uploads_middleware.py`:
  looks for a sibling `.md` file produced by the conversion pipeline
- Update `UploadsMiddleware._create_files_message()` to render the outline
  under each file entry with `L{line}: {title}` format and a `read_file`
  prompt for range-based reading
- Tests: 10 new tests for `extract_outline()`, 4 new tests for outline
  injection in `UploadsMiddleware`; existing test updated for new `outline`
  field in `uploaded_files` state

Partially addresses #1647 (agent ignores uploaded files).

* fix(uploads): stream outline file reads and strip inline bold from heading titles

- Switch extract_outline() from read_text().splitlines() to open()+line iteration
  so large converted documents are not loaded into memory on every agent turn;
  exits as soon as MAX_OUTLINE_ENTRIES is reached (Copilot suggestion)
- Strip **...** wrapper from standard Markdown heading titles before appending
  to outline so agent context stays clean (e.g. "## **Overview**" → "Overview")
  (Copilot suggestion)
- Remove unused pathlib.Path import and fix import sort order in test_file_conversion.py
  to satisfy ruff CI lint

* fix(uploads): show truncation hint when outline exceeds MAX_OUTLINE_ENTRIES

When extract_outline() hits the cap it now appends a sentinel entry
{"truncated": True} instead of silently dropping the rest of the headings.
UploadsMiddleware reads the sentinel and renders a hint line:

  ... (showing first 50 headings; use `read_file` to explore further)

Without this the agent had no way to know the outline was incomplete and
would treat the first 50 headings as the full document structure.

* fix(uploads): fall back to configurable.thread_id when runtime.context lacks thread_id

runtime.context does not always carry thread_id (depends on LangGraph
invocation path). ThreadDataMiddleware already falls back to
get_config().configurable.thread_id — apply the same pattern so
UploadsMiddleware can resolve the uploads directory and attach outlines
in all invocation paths.

* style: apply ruff format

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-03 20:52:47 +08:00
d 🔹
6dbdd4674f
fix: guarantee END sentinel delivery when stream bridge queue is full (#1695)
When MemoryStreamBridge queue reaches capacity, publish_end() previously
used the same 30s timeout + drop strategy as regular events. If the END
sentinel was dropped, subscribe() would loop forever waiting for it,
causing the SSE connection to hang indefinitely and leaking _queues and
_counters resources for that run_id.

Changes:
- publish_end() now evicts oldest regular events when queue is full to
  guarantee END sentinel delivery — the sentinel is the only signal that
  allows subscribers to terminate
- Added per-run drop counters (_dropped_counts) with dropped_count() and
  dropped_total properties for observability
- cleanup() and close() now clear drop counters
- publish() logs total dropped count per run for easier debugging

Tests:
- test_end_sentinel_delivered_when_queue_full: verifies END arrives even
  with a completely full queue
- test_end_sentinel_evicts_oldest_events: verifies eviction behavior
- test_end_sentinel_no_eviction_when_space_available: no side effects
  when queue has room
- test_concurrent_tasks_end_sentinel: 4 concurrent producer/consumer
  pairs all terminate properly
- test_dropped_count_tracking, test_dropped_total,
  test_cleanup_clears_dropped_counts, test_close_clears_dropped_counts:
  drop counter coverage

Closes #1689

Co-authored-by: voidborne-d <voidborne-d@users.noreply.github.com>
2026-04-03 20:12:30 +08:00
Octopus
83039fa22c
fix: use SystemMessage+HumanMessage for follow-up question generation (#1751)
* fix: use SystemMessage+HumanMessage for follow-up question generation (fixes #1697)

Some models (e.g. MiniMax-M2.7) require the system prompt and user
content to be passed as separate message objects rather than a single
combined string. Invoking with a plain string sends everything as a
HumanMessage, which causes these models to ignore the generation
instructions and fail to produce valid follow-up questions.

* test: verify model is invoked with SystemMessage and HumanMessage
2026-04-03 20:09:01 +08:00
finallylly
1694c616ef
feat(sandbox): add read-only support for local sandbox path mappings (#1808) 2026-04-03 19:46:22 +08:00
DanielWalnut
c6cdf200ce
feat(sandbox): add built-in grep and glob tools (#1784)
* feat(sandbox): add grep and glob tools

* refactor(aio-sandbox): use native file search APIs

* fix(sandbox): address review issues in grep/glob tools

- aio_sandbox: use should_ignore_path() instead of should_ignore_name()
  for include_dirs=True branch to filter nested ignored paths correctly
- aio_sandbox: add early exit when max_results reached in glob loop
- aio_sandbox: guard entry.path.startswith(path) before stripping prefix
- aio_sandbox: validate regex locally before sending to remote API
- search: skip lines exceeding max_line_chars to prevent ReDoS
- search: remove resolve() syscall in os.walk loop
- tools: avoid double get_thread_data() call in glob_tool/grep_tool
- tests: add 6 new cases covering the above code paths
- tests: patch get_app_config in truncation test to isolate config

* Fix sandbox grep/glob review feedback

* Remove unrelated Langfuse RFC from PR
2026-04-03 16:03:06 +08:00
Admire
48565664e0
fix ACP mcpServers payload (#1735)
* fix ACP mcpServers payload

* Handle invalid ACP MCP config
2026-04-03 15:28:56 +08:00
knukn
76fad8b08d
feat(client): add available_skills parameter to DeerFlowClient (#1779)
* feat(client): add `available_skills` parameter to DeerFlowClient for dynamic runtime skill filtering

* Update backend/packages/harness/deerflow/client.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(client): include `agent_name` and `available_skills` in agent config cache key

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-03 11:22:58 +08:00
ppyt
5664b9d413
fix: inject longTermBackground into memory prompt (#1734)
The format_memory_for_injection function only processed recentMonths and
earlierContext from the history section, silently dropping longTermBackground.

The LLM writes longTermBackground correctly and it persists to memory.json,
but it was never injected into the system prompt — making the user's
long-term background invisible to the AI.

Add the missing field handling and a regression test.

Co-authored-by: ppyt <14163465+ppyt@users.noreply.github.com>
2026-04-03 11:21:58 +08:00
Subham Singhania
6de9c7b43f
Improve Python reliability in channel retries and thread typing (#1776)
Agent-Logs-Url: https://github.com/0xxy0/deer-flow/sessions/95336da6-e16d-43b4-834a-e5534c9396c5

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-04-03 07:50:11 +08:00
moose-lab
f56d0b4869
fix(sandbox): exclude URL paths from absolute path validation (#1385) (#1419)
* fix(sandbox): URL路径被误判为不安全绝对路径 (#1385)

在本地沙箱模式下,bash工具对命令做绝对路径安全校验时,会把curl命令中的
HTTPS URL(如 https://example.com/api/v1/check)误识别为本地绝对路径并拦截。

根因:_ABSOLUTE_PATH_PATTERN 正则的负向后行断言 (?<![:\w]) 只排除了冒号和
单词字符,但 :// 中第二个斜杠前面是第一个斜杠(/),不在排除列表中,导致
//example.com/api/... 被匹配为绝对路径 /example.com/api/...。

修复:在负向后行断言中增加斜杠字符,改为 (?<![:\w/]),使得 :// 中的连续
斜杠不会触发绝对路径匹配。同时补充了URL相关的单元测试用例。

Signed-off-by: moose-lab <moose-lab@users.noreply.github.com>

* fix(sandbox): refine absolute path regex to preserve file:// defense-in-depth

Change lookbehind from (?<![:\w/]) to (?<![:\w])(?<!:/) so only the
second slash in :// sequences is excluded. This keeps URL paths from
false-positiving while still letting the regex detect /etc/passwd in
file:///etc/passwd. Also add explicit file:// URL blocking and tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: moose-lab <moose-lab@users.noreply.github.com>
Co-authored-by: moose-lab <moose-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-02 16:09:14 +08:00
Varian_米泽
a2cb38f62b
fix: prevent concurrent subagent file write conflicts in sandbox tools (#1714)
* fix: prevent concurrent subagent file write conflicts

Serialize same-path str_replace operations in sandbox tools

Guard AioSandbox write_file/update_file with the existing sandbox lock

Add regression tests for concurrent str_replace and append races

Verify with backend full tests and ruff lint checks

* fix(sandbox): Fix the concurrency issue of file operations on the same path in isolated sandboxes.

Ensure that different sandbox instances use independent locks for file operations on the same virtual path to avoid concurrency conflicts. Change the lock key from a single path to a composite key of (sandbox.id, path), and add tests to verify the concurrent safety of isolated sandboxes.

* feat(sandbox): Extract file operation lock logic to standalone module and fix concurrency issues

Extract file operation lock related logic from tools.py into a separate file_operation_lock.py module.
Fix data race issues during concurrent str_replace and write_file operations.
2026-04-02 15:39:41 +08:00
knukn
f8fb8d6fb1
feat/per agent skill filter (#1650)
* feat(agent): 为AgentConfig添加skills字段并更新lead_agent系统提示

在AgentConfig中添加skills字段以支持配置agent可用技能
更新lead_agent的系统提示模板以包含可用技能信息

* fix: resolve agent skill configuration edge cases and add tests

* Update backend/packages/harness/deerflow/agents/lead_agent/prompt.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* refactor(agent): address PR review comments for skills configuration

- Add detailed docstring to `skills` field in `AgentConfig` to clarify the semantics of `None` vs `[]`.
- Add unit tests in `test_custom_agent.py` to verify `load_agent_config()` correctly parses omitted skills and explicit empty lists.
- Fix `test_make_lead_agent_empty_skills_passed_correctly` to include `agent_name` in the runtime config, ensuring it exercises the real code path.

* docs: 添加关于按代理过滤技能的配置说明

在配置示例文件和文档中添加说明,解释如何通过代理的config.yaml文件限制加载的技能

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-02 15:02:09 +08:00
totoyang
2d1f90d5dc
feat(tracing): add optional Langfuse support (#1717)
* feat(tracing): add optional Langfuse support

* Fix tracing fail-fast behavior for explicitly enabled providers

* fix(lint)
2026-04-02 13:06:10 +08:00
3a672b39c7
Fix/1681 llm call retry handling (#1683)
* fix(runtime): handle llm call errors gracefully

* fix(runtime): preserve graph control flow in llm retry middleware

---------

Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>
2026-04-02 10:12:17 +08:00
SHIYAO ZHANG
df5339b5d0
feat(sandbox): truncate oversized bash and read_file tool outputs (#1677)
* feat(sandbox): truncate oversized bash and read_file tool outputs

Long tool outputs (large directory listings, multi-MB source files) can
overflow the model's context window. Two new configurable limits:

- bash_output_max_chars (default 20000): middle-truncates bash output,
  preserving both head and tail so stderr at the end is not lost
- read_file_output_max_chars (default 50000): head-truncates file output
  with a hint to use start_line/end_line for targeted reads

Both limits are enforced at the tool layer (sandbox/tools.py) rather
than middleware, so truncation is guaranteed regardless of call path.
Setting either limit to 0 disables truncation entirely.

Measured: read_file on a 250KB source file drops from 63,698 tokens to
19,927 tokens (69% reduction) with the default limit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(tests): remove unused pytest import and fix import sort order

* style: apply ruff format to sandbox/tools.py

* refactor(sandbox): address Copilot review feedback on truncation feature

- strict hard cap: while-loop ensures result (including marker) ≤ max_chars
- max_chars=0 now returns "" instead of original output
- get_app_config() wrapped in try/except with fallback to defaults
- sandbox_config.py: add ge=0 validation on truncation limit fields
- config.example.yaml: bump config_version 4→5
- tests: add len(result) <= max_chars assertions, edge-case (max=0, small
  max, various sizes) tests; fix skipped-count test for strict hard cap

* refactor(sandbox): replace while-loop truncation with fixed marker budget

Use a pre-allocated constant (_MARKER_MAX_LEN) instead of a convergence
loop to ensure result <= max_chars. Simpler, safer, and skipped-char
count in the marker is now an exact predictable value.

* refactor(sandbox): compute marker budget dynamically instead of hardcoding

* fix(sandbox): make max_chars=0 disable truncation instead of returning empty string

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: JeffJiang <for-eleven@hotmail.com>
2026-04-02 09:22:41 +08:00
LYU Yichen
0a379602b8
fix: avoid treating Feishu file paths as commands (#1654)
Feishu channel classified any slash-prefixed text (including absolute
paths such as /mnt/user-data/...) as a COMMAND, causing them to be
misrouted through the command pipeline instead of the chat pipeline.

Fix by introducing a shared KNOWN_CHANNEL_COMMANDS frozenset in
app/channels/commands.py — the single authoritative source for the set
of supported slash commands.  Both the Feishu inbound parser and the
ChannelManager's unknown-command reply now derive from it, so adding
or removing a command requires only one edit.

Changes:
- app/channels/commands.py (new): defines KNOWN_CHANNEL_COMMANDS
- app/channels/feishu.py: replace local KNOWN_FEISHU_COMMANDS with the
  shared constant; _is_feishu_command() now gates on it
- app/channels/manager.py: import KNOWN_CHANNEL_COMMANDS and use it in
  the unknown-command fallback reply so the displayed list stays in sync
- tests/test_feishu_parser.py: parametrize over every entry in
  KNOWN_CHANNEL_COMMANDS (each must yield msg_type=command) and add
  parametrized chat cases for /unknown, absolute paths, etc.

Made with Cursor

Made-with: Cursor

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-01 23:23:00 +08:00
Jason
1fb5acee39
fix(gateway): prevent 400 error when client sends context with configurable (#1660)
* fix(gateway): prevent 400 error when client sends context with configurable

Fixes #1290

LangGraph >= 0.6.0 rejects requests that include both 'configurable' and
'context' in the run config. If the client (e.g. useStream hook) sends
a 'context' key, we now honour it and skip creating our own
'configurable' dict to avoid the conflict.

When no 'context' is provided, we fall back to the existing
'configurable' behaviour with thread_id.

* fix(gateway): address review feedback — warn on dual keys, fix runtime injection, add tests

- Log a warning when client sends both 'context' and 'configurable' so
  it's no longer silently dropped (reviewer feedback)
- Ensure thread_id is available in config['context'] when present so
  middlewares can find it there too
- Add test coverage for the context path, the both-keys-present case,
  passthrough of other keys, and the no-config fallback

* style: ruff format services.py

---------

Co-authored-by: JasonOA888 <JasonOA888@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-01 23:21:32 +08:00
Alian
e97c8c9943
fix(skills): support parsing multiline YAML strings in SKILL.md frontmatter (#1703)
* fix(skills): support parsing multiline YAML strings in SKILL.md frontmatter

* test(skills): add tests for multiline YAML descriptions
2026-04-01 23:08:30 +08:00
rayhpeng
c2ff59a5b1
fix(gateway): merge context field into configurable for langgraph-compat runs (#1699) (#1707)
The langgraph-compat layer dropped the DeerFlow-specific `context` field
from run requests, causing agent config (subagent_enabled, is_plan_mode,
thinking_enabled, etc.) to fall back to defaults. Add `context` to
RunCreateRequest and merge allowlisted keys into config.configurable in
start_run, with existing configurable values taking precedence.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 17:17:09 +08:00
Shengyuan Wang
2f3744f807
refactor: replace sync requests with async httpx in Jina AI client (#1603)
* refactor: replace sync requests with async httpx in Jina AI client

Replace synchronous `requests.post()` with `httpx.AsyncClient` in
JinaClient.crawl() and make web_fetch_tool async. This is part of the
planned async concurrency optimization for the agent hot path
(see docs/TODO.md).

* fix: address Copilot review feedback on async Jina client

- Short-circuit error strings in web_fetch_tool before passing to
  ReadabilityExtractor, preventing misleading extraction results
- Log missing JINA_API_KEY warning only once per process to reduce
  noise under concurrent async fetching
- Use logger.exception instead of logger.error in crawl exception
  handler to preserve stack traces for debugging
- Add async web_fetch_tool tests and warn-once coverage

* fix: mock get_app_config in web_fetch_tool tests for CI

The web_fetch_tool tests failed in CI because get_app_config requires
a config.yaml file that isn't present in the test environment. Mock
the config loader to remove the filesystem dependency.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-01 17:02:39 +08:00
AochenShen99
0cdecf7b30
feat(memory): structured reflection + correction detection in MemoryMiddleware (#1620) (#1668)
* feat(memory): add structured reflection and correction detection

* fix(memory): align sourceError schema and prompt guidance

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-01 16:45:29 +08:00
LYU Yichen
3e461d9d08
fix: use safe docker bind mount syntax for sandbox mounts (#1655)
Docker's -v host:container syntax is ambiguous for Windows drive-letter
paths (e.g. D:/...) because ':' is both the drive separator and the
volume separator, causing mount failures on Windows hosts.

Introduce _format_container_mount() which uses '--mount type=bind,...'
for Docker (unambiguous on all platforms) and keeps '-v' for Apple
Container runtime which does not support the --mount flag yet.

Adds unit tests covering Windows paths, read-only mounts, and Apple
Container pass-through.

Made-with: Cursor
2026-04-01 11:42:12 +08:00
d 🔹
6ff60f2af1
fix(gateway): forward assistant_id as agent_name in build_run_config (#1667)
* fix(gateway): forward assistant_id as agent_name in build_run_config

Fixes #1644

When the LangGraph Platform-compatible /runs endpoint receives a custom
assistant_id (e.g. 'finalis'), the Gateway's build_run_config() silently
ignored it — configurable['agent_name'] was never set, so make_lead_agent
fell through to the default lead agent and SOUL.md was never loaded.

Root cause (introduced in #1403):
  resolve_agent_factory() correctly falls back to make_lead_agent for all
  assistant_id values, but build_run_config() had no assistant_id parameter
  and never injected configurable['agent_name'].  The full call chain:

    POST /runs (assistant_id='finalis')
      → resolve_agent_factory('finalis')   # returns make_lead_agent ✓
      → build_run_config(thread_id, ...)   # no agent_name injected ✗
        → make_lead_agent(config)
          → cfg.get('agent_name') → None
            → load_agent_soul(None) → base SOUL.md (doesn't exist) → None

Fix:
- Add keyword-only  parameter to build_run_config().
- When assistant_id is set and differs from 'lead_agent', inject it as
  configurable['agent_name'] (matching the channel manager's existing
  _resolve_run_params() logic for IM channels).
- Honour an explicit configurable['agent_name'] in the request body;
  assistant_id mapping only fills the gap when it is absent.
- Remove stale log-only branch from resolve_agent_factory(); update
  docstring to explain the factory/configurable split.

Tests added (test_gateway_services.py):
- Custom assistant_id injects configurable['agent_name']
- 'lead_agent' assistant_id does NOT inject agent_name
- None assistant_id does NOT inject agent_name
- Explicit configurable['agent_name'] in request is not overwritten
- resolve_agent_factory returns make_lead_agent for all inputs

* style: format with ruff

* fix: validate and normalize assistant_id to prevent path traversal

Addresses Copilot review: strip/lowercase/replace underscores and
reject names that don't match [a-z0-9-]+, consistent with
ChannelManager._normalize_custom_agent_name().

---------

Co-authored-by: voidborne-d <voidborne-d@users.noreply.github.com>
2026-04-01 11:15:56 +08:00
Matt Van Horn
a3bfea631c
fix(sandbox): serialize concurrent exec_command calls in AioSandbox (#1435)
* fix(sandbox): serialize concurrent exec_command calls in AioSandbox

The AIO sandbox container maintains a single persistent shell session
that corrupts when multiple exec_command requests arrive concurrently
(e.g. when ToolNode issues parallel tool_calls). The corrupted session
returns 'ErrorObservation' strings as output, cascading into subsequent
commands.

Add a threading.Lock to AioSandbox to serialize shell commands. As a
secondary defense, detect ErrorObservation in output and retry with a
fresh session ID.

Fixes #1433

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(sandbox): address Copilot review findings

- Fix shell injection in list_dir: use shlex.quote(path) to escape
  user-provided paths in the find command
- Narrow ErrorObservation retry condition from broad substring match
  to the specific corruption signature to prevent false retries
- Improve test_lock_prevents_concurrent_execution: use threading.Barrier
  to ensure all workers contend for the lock simultaneously
- Improve test_list_dir_uses_lock: assert lock.locked() is True during
  exec_command to verify lock acquisition

* style: auto-format with ruff

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-31 22:33:35 +08:00
Admire
aae59a8ba8
fix: surface configured sandbox mounts to agents (#1638)
* fix: surface configured sandbox mounts to agents

* fix: address PR review feedback

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-31 22:22:30 +08:00
Admire
3ff15423d6
fix Windows Docker sandbox path mounting (#1634)
* fix windows docker sandbox paths

* fix windows sandbox mount validation

* fix backend checks for windows sandbox path PR
2026-03-31 22:19:27 +08:00