1872 Commits

Author SHA1 Message Date
rayhpeng
a5831d3abf Merge branch 'main' into rayhpeng/persistence-scaffold
# Conflicts:
#	backend/tests/test_model_factory.py
2026-04-06 17:11:49 +08:00
Zhou
1ced6e977c
fix(backend): preserve viewed image reducer metadata (#1900)
Fix concurrent viewed_images state updates for multi-image input by preserving the reducer metadata in the vision middleware state schema.
2026-04-06 16:47:19 +08:00
Zhou
f5088ed70d
fix(frontend): artifact download action bounds and lint errors (#1899)
* fix: keep artifact download action in bounds

* fix: fix lint error
2026-04-06 16:34:40 +08:00
Zhou
55e78de6fc
fix: wrap suggestion chips without overlapping input (#1895)
* fix: wrap suggestion chips without overlapping input

* fix: fix lint error
2026-04-06 16:30:57 +08:00
NmanQAQ
dd30e609f7
feat(models): add vLLM provider support (#1860)
support for vLLM 0.19.0 OpenAI-compatible chat endpoints and fixes the Qwen reasoning toggle so flash mode can actually disable thinking.

Co-authored-by: NmanQAQ <normangyao@qq.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-06 15:18:34 +08:00
yangzheli
5fd2c581f6
fix: add output truncation to ls_tool to prevent context window overflow (#1896)
ls_tool was the only sandbox tool without output size limits, allowing
multi-MB results from large directories to blow up the model context
window. Add head-truncation (configurable via ls_output_max_chars,
default 20000) consistent with existing bash and read_file truncation.

Closes #1887

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-06 15:09:57 +08:00
Chincherry93
d7a3eff23e
fix(docker): command syntax for LANGGRAPH_ALLOW_BLOCKING (#1891) 2026-04-06 15:02:29 +08:00
qqwas
ee06440205
fix(frontend): Update route.ts default backend port(#1892) 2026-04-06 14:54:50 +08:00
7c68dd4ad4
Fix(#1702): stream resume run (#1858)
* fix: repair stream resume run metadata

# Conflicts:
#	backend/packages/harness/deerflow/runtime/stream_bridge/memory.py
#	frontend/src/core/threads/hooks.ts

* fix(stream): repair resumable replay validation

---------

Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-06 14:51:10 +08:00
rayhpeng
ddd8613520 Merge remote-tracking branch 'origin/rayhpeng/persistence-scaffold' into rayhpeng/persistence-scaffold 2026-04-06 11:44:42 +08:00
rayhpeng
d592a98452 docs: annotate DbRunEventStore.put() as low-frequency path
Add docstring clarifying that put() opens a per-call transaction with
FOR UPDATE and should only be used for infrequent writes (currently
just the initial human_message event). High-throughput callers should
use put_batch() instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 11:24:29 +08:00
rayhpeng
0af0ae7fbb perf: use SQL aggregation for feedback stats and thread token usage
Replace Python-side counting in FeedbackRepository.aggregate_by_run with
a single SELECT COUNT/SUM query. Add RunStore.aggregate_tokens_by_thread
abstract method with SQL GROUP BY implementation in RunRepository and
Python fallback in MemoryRunStore. Simplify the thread_token_usage
endpoint to delegate to the new method, eliminating the limit=10000
truncation risk.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 11:20:34 +08:00
rayhpeng
332fb18b34 refactor(gateway): move sanitize_log_param to app/gateway/utils.py
Extract the log-injection sanitizer from routers/threads.py into a shared
utils module and rename to sanitize_log_param (public API). Eliminates the
reverse service → router import in services.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 11:09:42 +08:00
rayhpeng
eba6810a44 refactor(runtime): introduce RunContext to reduce run_agent parameter bloat
Extract checkpointer, store, event_store, run_events_config, thread_meta_repo,
and follow_up_to_run_id into a frozen RunContext dataclass. Add get_run_context()
in deps.py to build the base context from app.state singletons. start_run() uses
dataclasses.replace() to enrich per-run fields before passing ctx to run_agent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 10:59:47 +08:00
rayhpeng
e4e4320af5
Merge branch 'main' into rayhpeng/persistence-scaffold 2026-04-06 10:22:53 +08:00
suyua9
29575c32f9
fix: expose custom events from DeerFlowClient.stream() (#1827)
* fix: expose custom client stream events

Signed-off-by: suyua9 <1521777066@qq.com>

* fix(client): normalize streamed custom mode values

* test(client): satisfy backend ruff import ordering

---------

Signed-off-by: suyua9 <1521777066@qq.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-06 10:09:39 +08:00
amonduuuul
ed90a2ee9d
fix(docker): recover invalid .venv to prevent startup restart loops (#1871)
* fix(docker): recover invalid .venv before service startup

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-06 08:34:25 +08:00
Willem Jiang
993fb0ff9d
fix: escape shell variables in production langgraph command (#1877) (#1880)
Escape  shell variables to prevent Docker Compose from attempting
substitution at parse time. Rename allow_blocking_flag to allow_blocking
for consistency with dev version.

Fixes the 'allow_blocking_flag not set' warning and enables --allow-blocking
flag to work correctly.
2026-04-06 08:24:51 +08:00
rayhpeng
8746a2bcd9 Merge remote-tracking branch 'origin/rayhpeng/persistence-scaffold' into rayhpeng/persistence-scaffold 2026-04-05 23:51:36 +08:00
rayhpeng
3f00a22df3
Potential fix for pull request finding 'Statement has no effect'
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-04-05 23:46:35 +08:00
rayhpeng
07954cf9d2 style: apply ruff format to persistence and runtime files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 23:44:48 +08:00
rayhpeng
107b3143c3 Merge branch 'main' into rayhpeng/persistence-scaffold
# Conflicts:
#	backend/Dockerfile
#	backend/uv.lock
2026-04-05 23:40:49 +08:00
rayhpeng
b94383c93a fix(persistence): address 22 review comments from CodeQL, Copilot, and Code Quality
Bug fixes:
- Sanitize log params to prevent log injection (CodeQL)
- Reset threads_meta.status to idle/error when run completes
- Attach messages only to latest checkpoint in /history response
- Write threads_meta on POST /threads so new threads appear in search

Lint fixes:
- Remove unused imports (journal.py, migrations/env.py, test_converters.py)
- Convert lambda to named function (engine.py, Ruff E731)
- Remove unused logger definitions in repos (Ruff F841)
- Add logging to JSONL decode errors and empty except blocks
- Separate assert side-effects in tests (CodeQL)
- Remove unused local variables in tests (Ruff F841)
- Fix max_trace_content truncation to use byte length, not char length

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 22:49:26 +08:00
rayhpeng
32f69674a5 chore: update uv.lock
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 22:12:41 +08:00
rayhpeng
fc4e3a52d4 fix(persistence): address review feedback on PR #1851
- Fix naive datetime.now() → datetime.now(UTC) in all ORM models
- Fix seq race condition in DbRunEventStore.put() with FOR UPDATE
  and UNIQUE(thread_id, seq) constraint
- Encapsulate _store access in RunManager.update_run_completion()
- Deduplicate _store.put() logic in RunManager via _persist_to_store()
- Add update_run_completion to RunStore ABC + MemoryRunStore
- Wire follow_up_to_run_id through the full create path
- Add error recovery to RunJournal._flush_sync() lost-event scenario
- Add migration note for search_threads breaking change
- Fix test_checkpointer_none_fix mock to set database=None

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 22:02:50 +08:00
rayhpeng
7fdf9cad99 fix: resolve merge conflict in .env.example
Keep both DATABASE_URL (from persistence-scaffold) and WECOM
credentials (from main) after the merge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 21:52:05 +08:00
greatmengqi
ca2fb95ee6
feat: unified serve.sh with gateway mode support (#1847) 2026-04-05 21:07:35 +08:00
Chris Z
117fa9b05d
fix(channels): normalize slack allowed user ids (#1802)
* fix(channels): normalize slack allowed user ids

* style(channels): apply backend formatter

---------

Co-authored-by: haimingZZ <15558128926@qq.com>
Co-authored-by: suyua9 <1521777066@qq.com>
2026-04-05 18:04:21 +08:00
28474c47cb
fix: avoid command palette hydration mismatch on macOS (#1563)
# Conflicts:
#	frontend/src/components/workspace/command-palette.tsx

Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-05 16:35:33 +08:00
thefoolgy
8049785de6
fix(memory): case-insensitive fact deduplication and positive reinforcement detection (#1804)
* fix(memory): case-insensitive fact deduplication and positive reinforcement detection

Two fixes to the memory system:

1. _fact_content_key() now lowercases content before comparison, preventing
   semantically duplicate facts like "User prefers Python" and "user prefers
   python" from being stored separately.

2. Adds detect_reinforcement() to MemoryMiddleware (closes #1719), mirroring
   detect_correction(). When users signal approval ("yes exactly", "perfect",
   "完全正确", etc.), the memory updater now receives reinforcement_detected=True
   and injects a hint prompting the LLM to record confirmed preferences and
   behaviors with high confidence.

   Changes across the full signal path:
   - memory_middleware.py: _REINFORCEMENT_PATTERNS + detect_reinforcement()
   - queue.py: reinforcement_detected field in ConversationContext and add()
   - updater.py: reinforcement_detected param in update_memory() and
     update_memory_from_conversation(); builds reinforcement_hint alongside
     the existing correction_hint

Tests: 11 new tests covering deduplication, hint injection, and signal
detection (Chinese + English patterns, window boundary, conflict with correction).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(memory): address Copilot review comments on reinforcement detection

- Tighten _REINFORCEMENT_PATTERNS: remove 很好, require punctuation/end-of-string boundaries on remaining patterns, split this-is-good into stricter variants
- Suppress reinforcement_detected when correction_detected is true to avoid mixed-signal noise
- Use casefold() instead of lower() for Unicode-aware fact deduplication
- Add missing test coverage for reinforcement_detected OR merge and forwarding in queue

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-05 16:23:00 +08:00
Evan Wu
9ca68ffaaa
fix: preserve virtual path separator style (#1828)
* fix: preserve virtual path separator style

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-05 15:52:22 +08:00
Markus Corazzione
0ffe5a73c1
chroe(config):Increase subagent max-turn limits (#1852) 2026-04-05 15:41:00 +08:00
Echo-Nie
d3b59a7931
docs: fix some broken links (#1864)
* Rename BACKEND_TODO.md to TODO.md in documentation

* Update MCP Setup Guide link in CONTRIBUTING.md

* Update reference to config.yaml path in documentation

* Fix config file path in TITLE_GENERATION_IMPLEMENTATION.md

Updated the path to the example config file in the documentation.
2026-04-05 15:35:42 +08:00
yangzheli
e5416b539a
fix(docker): use multi-stage build to remove build-essential from runtime image (#1846)
* fix(docker): use multi-stage build to remove build-essential from runtime image

The build-essential toolchain (~200 MB) was only needed for compiling
native Python extensions during `uv sync` but remained in the final
image, increasing size and attack surface. Split the Dockerfile into
a builder stage (with build-essential) and a clean runtime stage that
copies only the compiled artifacts, Node.js, Docker CLI, and uv.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(docker): add dev stage and pin docker:cli per review feedback

Address Copilot review comments:
- Add a `dev` build stage (FROM builder) that retains build-essential
  so startup-time `uv sync` in dev containers can compile from source
- Update docker-compose-dev.yaml to use `target: dev` for gateway and
  langgraph services
- Keep the clean runtime stage (no build-essential) as the default
  final stage for production builds

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-05 15:30:34 +08:00
SHIYAO ZHANG
72d4347adb
fix(sandbox): guard against None runtime.context in sandbox tool helpers (#1853)
sandbox_from_runtime() and ensure_sandbox_initialized() write
sandbox_id into runtime.context after acquiring a sandbox. When
lazy_init=True and no context is supplied to the graph run,
runtime.context is None (the LangGraph default), causing a TypeError
on the assignment.

Add `if runtime.context is not None` guards at all three write sites.
Reads already had equivalent guards (e.g. `runtime.context.get(...) if
runtime.context else None`); this brings writes into line.
2026-04-05 10:58:38 +08:00
Octopus
a283d4a02d
fix: include soul field in GET /api/agents list response (fixes #1819) (#1863)
Previously, the list endpoint always returned soul=null because
_agent_config_to_response() was called without include_soul=True.
This caused confusion since PUT /api/agents/{name} and GET /api/agents/{name}
both returned the soul content, but the list endpoint silently omitted it.

Co-authored-by: octo-patch <octo-patch@users.noreply.github.com>
2026-04-05 10:49:58 +08:00
yangzheli
5f8dac66e6
chore(deps): update uv.lock (#1848)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-05 10:22:14 +08:00
Adem Akdoğan
8bb14fa1a7
feat(skills): add academic-paper-review, code-documentation, and newsletter-generation skills (#1861)
Add three new public skills to enhance DeerFlow's content creation capabilities:

- **academic-paper-review**: Structured peer-review-quality analysis of
  research papers following top-venue review standards (NeurIPS, ICML, ACL).
  Covers methodology assessment, contribution evaluation, literature
  positioning, and constructive feedback with a 3-phase workflow.

- **code-documentation**: Professional documentation generation for software
  projects, including README generation, API reference docs, architecture
  documentation with Mermaid diagrams, and inline code documentation
  supporting Python, TypeScript, Go, Rust, and Java conventions.

- **newsletter-generation**: Curated newsletter creation with research
  workflow, supporting daily digest, weekly roundup, deep-dive, and industry
  briefing formats. Includes audience-specific tone adaptation and
  multi-source content curation.

All skills:
- Follow the existing SKILL.md frontmatter convention (name + description)
- Pass the official _validate_skill_frontmatter() validation
- Use hyphen-case naming consistent with existing skills
- Contain only allowed frontmatter properties
- Include comprehensive examples, quality checklists, and output templates
2026-04-05 10:19:35 +08:00
rayhpeng
8a6ed365aa fix(middleware): pass tagged config to TitleMiddleware ainvoke call
Without the config, the middleware:title tag was not injected,
causing the LLM response to be recorded as a lead_agent ai_message
in run_events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 21:43:04 +08:00
rayhpeng
cef83878d4 fix: remove duplicate optional-dependencies header in pyproject.toml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 21:34:36 +08:00
rayhpeng
4737fc3aa9 Merge branch 'main' into rayhpeng/persistence-scaffold
# Conflicts:
#	.env.example
#	backend/packages/harness/deerflow/agents/middlewares/title_middleware.py
2026-04-04 21:28:07 +08:00
rayhpeng
b55a9c8d28 feat(threads): history endpoint reads messages from event store
- POST /api/threads/{thread_id}/history now combines two data sources:
  checkpointer for checkpoint_id, metadata, title, thread_data;
  event store for messages (complete history, not truncated by summarization)
- Strip internal LangGraph metadata keys from response
- Remove full channel_values serialization in favor of selective fields

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 21:23:32 +08:00
DanielWalnut
2a150f5d4a
fix: unblock concurrent threads and workspace hydration (#1839)
* fix: unblock concurrent threads and workspace hydration

* fix: restore async title generation

* fix: address PR review feedback

* style: format lead agent prompt
2026-04-04 21:19:35 +08:00
rayhpeng
35001c7c73 feat(threads): switch search endpoint to threads_meta table and sync title
- POST /api/threads/search now queries threads_meta table directly,
  removing the two-phase Store + Checkpointer scan approach
- Add ThreadMetaRepository.search() with metadata/status filters
- Add ThreadMetaRepository.update_display_name() for title sync
- Worker syncs checkpoint title to threads_meta.display_name on run completion
- Map display_name to values.title in search response for API compatibility

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 21:07:21 +08:00
rayhpeng
52e7acafee feat(events): align message events with checkpoint format and add middleware tag injection
- Message events (ai_message, ai_tool_call, tool_result, human_message) now use
  BaseMessage.model_dump() format, matching LangGraph checkpoint values.messages
- on_tool_end extracts tool_call_id/name/status from ToolMessage objects
- on_tool_error now emits tool_result message events with error status
- record_middleware uses middleware:{tag} event_type and middleware category
- Summarization custom events use middleware:summarize category
- TitleMiddleware injects middleware:title tag via get_config() inheritance
- SummarizationMiddleware model bound with middleware:summarize tag
- Worker writes human_message using HumanMessage.model_dump()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 20:52:27 +08:00
luobo
1c0051c1db
fix(frontend): keep prompt attachments from breaking before upload (#1833)
* fix(frontend): preserve prompt attachment files during upload

* fix(frontend): harden prompt attachment fallback and tests

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-04 14:54:35 +08:00
luobo
144c9b2464
fix(frontend): block unsupported .app uploads (#1834)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-04 14:42:26 +08:00
SHIYAO ZHANG
163121d327
fix(uploads): handle split-bold headings and ** ** artefacts in extract_outline (#1838)
* feat(uploads): guide agent to use grep/glob/read_file for uploaded documents

Add workflow guidance to the <uploaded_files> context block so the agent
knows to use grep and glob (added in #1784) alongside read_file when
working with uploaded documents, rather than falling back to web search.

This is the final piece of the three-PR PDF agentic search pipeline:
- PR1 (#1727): pymupdf4llm converter produces structured Markdown with headings
- PR2 (#1738): document outline injected into agent context with line numbers
- PR3 (this):  agent guided to use outline + grep + read_file workflow

* feat(uploads): add file-first priority and fallback guidance to uploaded_files context

* fix(uploads): handle split-bold headings and ** ** artefacts in extract_outline

- Add _clean_bold_title() to merge adjacent bold spans (** **) produced
  by pymupdf4llm when bold text crosses span boundaries
- Add _SPLIT_BOLD_HEADING_RE (Style 3) to recognise **<num>** **<title>**
  headings common in academic papers; excludes pure-number table headers
  and rows with more than 4 bold blocks
- When outline is empty, read first 5 non-empty lines of the .md as a
  content preview and surface a grep hint in the agent context
- Update _format_file_entry to render the preview + grep hint instead of
  silently omitting the outline section
- Add 3 new extract_outline tests and 2 new middleware tests (65 total)

* fix(uploads): address Copilot review comments on extract_outline regex

- Replace ASCII [A-Za-z] guard with negative lookahead to support non-ASCII
  titles (e.g. **1** **概述**); pure-numeric/punctuation blocks still excluded
- Replace .+ with [^*]+ and cap repetition at {0,2} (four blocks total) to
  keep _SPLIT_BOLD_HEADING_RE linear and avoid ReDoS on malformed input
- Remove now-redundant len(blocks) <= 4 code-level check (enforced by regex)
- Log debug message with exc_info when preview extraction fails
2026-04-04 14:25:08 +08:00
fengxsong
19809800f1
feat: support wecom channel (#1390)
* feat: support wecom channel

* fix: sending file to client

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* test: add unit tests for wecom channel

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* docs: add example configs and setup docs

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* revert pypi default index setting

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* revert: keeping codes in harness untouched

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* fix: format issue

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* fix: resolve Copilot comments

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

---------

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-04 11:28:35 +08:00
Albert Zheng
6473d38917
fix(frontend): resolve button hydration mismatch with undefined variant/size (#1506)
Server-rendered data-variant={undefined} didn't match client hydration.
Now only render data-variant and data-size when explicitly set.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: JeffJiang <for-eleven@hotmail.com>
2026-04-04 11:21:04 +08:00