275 Commits

Author SHA1 Message Date
Michiel Voortman
e0cccd6b37
feat(sandbox): add boxd sandbox provider (#8314)
* feat(sandbox): add boxd sandbox provider

Adds deerflow.community.boxd_sandbox:BoxdSandboxProvider. Each user/thread
scope runs in an isolated boxd machine. New machines get the thread's files
and skills before they are handed out, release mirrors outputs back to
thread storage, and released machines wait in the shared warm pool.

The boxd SDK is an optional extra (deerflow-harness[boxd]). Unit tests run
against a fake client; tests/test_boxd_sandbox_live.py runs against real
machines when DEER_FLOW_RUN_LIVE_TESTS=1 and BOXD_API_KEY are set.

* fix(boxd): retain failed cleanup and bound output mirror calls

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-18 21:29:15 +08:00
Bas Cancrinus
f32570d539
feat(opensandbox): configurable sandbox resource limits and requests (#8305)
* feat(opensandbox): configurable sandbox resource limits and requests

The OpenSandbox provider passes no resources to SandboxSync.create(), so
every sandbox gets the SDK default of 1 CPU / 2Gi and the server's default
requests. Workloads that need more (builds, PDF rendering, data work) can't
get it, and clusters with tight quotas can't ask for less.

Add two optional sandbox options, `resource` (limits) and
`resource_requests`, each a map of resource names to quantities, forwarded
to SandboxSync.create(). Numbers become strings; other value types fail at
provider startup. Unset, null or empty keeps today's defaults.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(opensandbox): reject blank or non-positive resource quantities; document requests

Review follow-up for #8305:
- Reject blank or whitespace resource names and quantities, and numeric
  quantities that are zero, negative or non-finite, at provider startup
  instead of failing at acquire().
- Document that only Kubernetes runtimes apply resource_requests, that an
  omitted request defaults to its limit, and that requests must fit within
  the limits, including the SDK default 1 CPU / 2Gi when resource is unset.
- Bump config_version to 67 (example, chart values, chart README) and add a
  CHANGELOG entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: drop the CHANGELOG entries (generated in batch by maintainers)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:01:07 +08:00
Aurora_
c5d072af91
feat(subagents): add optional SQLite checkpoint history (#8265)
* feat(subagents): add optional SQLite checkpoint history

* docs(subagents): clarify checkpoint history limits

* fix(subagents): resolve checkpoint factory at startup

---------

Co-authored-by: Nan Gao <thunderggnn@gmail.com>
2026-11-16 13:29:26 +08:00
Totoro
b0bca9dcb4
feat(mcp): bind HTTP tool credentials to approved custom agents (#8242)
* feat(mcp): bind agent credentials to approved definitions

* fix(mcp): enforce live auth policy and pin task destinations

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-11-14 22:10:33 +08:00
Hyeonsang Cho
1804b7c522
fix(gateway): read the run and authorization config off the event loop (#8137)
* fix(gateway): read the run and authorization config off the event loop

get_app_config() stats and hashes config.yaml and extensions_config.json
on every call and reparses them after an edit. Run admission read it
inline on the loop two to four times per start (get_run_context,
build_run_config, the model allowlist, the approval switch, conversation
references), and the model list, model details, skill list, suggestions
and sandbox gates read the authorization section inline once each.

start_run now builds its run context in a worker and reuses that
context's app_config for the rest of the start, so one run start sees
one snapshot. Scheduled dispatch resolves its recursion limit in a
worker, run inputs take only the AgentRuns capability, and the
route-scoped authorization helpers run through asyncio.to_thread, as
/api/models/resolve already did.

* docs(changelog): reference #8137 for the off-loop run and authorization config reads

* fix(gateway): validate the tool approval mode against the run's config snapshot

The approval-mode check added on main read get_app_config() inline in
start_run, so every run start revalidated config.yaml on the event loop
again. It now uses the snapshot the run context was built with, like the
model allowlist and the approval switch.

* test(gateway): seed the skill catalog the authorization anchor lists

The skill listing anchor relied on the checkout's skills/public tree
containing academic-paper-review. It now writes one allowed and one
hidden skill under its own skills.path, so it passes without the repo
catalog and still fails if the policy is not applied.
2026-11-14 17:33:41 +08:00
ywy
98e8598adb
fix(config): delete thread dir through extended-length path to avoid WinError 145 (#8144)
Co-authored-by: Nan Gao <thunderggnn@gmail.com>
2026-11-12 14:24:24 +08:00
Bas Cancrinus
13e6fc29e5
feat(opensandbox): add opt-in secure_access sandbox option (#8162)
* feat(opensandbox): add opt-in secure_access sandbox option

Forward a new `sandbox.secure_access` option (default false) to
SandboxSync.create so OpenSandbox can require a per-sandbox endpoint
token at the ingress gateway and server proxy. The server returns the
token as an endpoint header and SDK 0.1.15 already attaches endpoint
headers to execd and file requests, so DeerFlow needs no token handling.

Non-boolean values are rejected at provider startup so a quoted "false"
cannot silently enable it. Documented in config.example.yaml,
CONFIGURATION.md, the SandboxConfig docstring and the provider README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(opensandbox): validate server proxy boolean

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Totoro <msy626836554@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-12 09:49:33 +08:00
Wenchao An
9699f04cdc
feat(projects): add visible project memory with extraction controls (#8117)
* feat(projects): add visible project memory with extraction controls

* fix(projects): avoid replaying committed memory extractions

* chore(ci): retrigger project memory checks

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-12 09:09:44 +08:00
Wenchao An
3d826c6d95
feat(approvals): add Ask, Auto and Full tool permission modes (#8033)
* feat(hitl): add human-in-the-loop approval for high-risk tool calls

* fix(hitl): address approval review findings

* docs(hitl): keep agent guidance under size limit

* fix(hitl): guard approval pause recovery

* docs(hitl): shrink inherited guidance

* fix(hitl): harden approval admission and recovery

* fix(approvals): address review safety and reconciliation gaps

* ci: retry approval checks after runner timeout

---------

Co-authored-by: wangzeren <1004695029@qq.com>
2026-11-11 20:59:05 +08:00
Wistom Zhang
218c64180e
feat(summarization): align built-in compaction prompt to preserve user requirements and approval conditions (#8114) (#8114)
In long-running agent workflows with repeated context compaction, empirical gateway ablations across 132 real sessions revealed that standard boundary compliance remains robust, while semantic drift can occur when model-generated summaries rephrase literal approval phrases and action conditions.

This change converges PR #8114 into a clean built-in summary prompt enhancement:
- Replaces regex extraction, tag elevation, and heuristic revocation with an extended built-in DeerFlow summary prompt (_DEERFLOW_SUMMARY_PROMPT).
- Instructs the summarizer to faithfully record active user requirements, literal approval phrases, permitted actions, and later explicit revisions without paraphrasing.
- Treats summary text strictly as historical data, fully preserving the security boundary and control/data plane separation of DurableContextMiddleware.
- Unifies automatic and manual compaction factory paths to select the built-in template when unconfigured, while preserving explicit summary_prompt precedence.
- Adds comprehensive sync/async unit tests covering built-in and custom templates.

Co-authored-by: Totoro <Totoro-qaq@users.noreply.github.com>
Co-authored-by: Totoro <msy626836554@gmail.com>
2026-11-11 19:36:32 +08:00
Xinmin Zeng
2a97ec3e58
docs: turn README into an index and move reference detail to docs/guide (#8113)
* docs: turn README into an index and move reference detail to docs/guide

The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md.

* docs: send user docs to topic guides and budget the root README

The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003).

* chore: remove unreferenced screenshots and finished plan notes

pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them.

* docs: route always-loaded AGENTS.md files to scoped guides

* docs(agents): cut scoped guides to scope, verify, traps, decisions

Each AGENTS.md keeps four kinds of content: the scope of the directory, the
verification commands, traps a local test run does not show, and decisions the
code cannot state. Values, field lists, function inventories, test names, issue
numbers and history are deleted, because the code, the tests and git state them.

Search and fetch providers share one community guide. Four guides with no
remaining content are removed. The size check now allows 6 KiB per file (hard
limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB).

* style: format frontend guide table and guidance test imports

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-11 13:26:17 +08:00
Zeren Wang
0d04322d41
feat(hitl): add human-in-the-loop approval for high-risk tool calls (#7979)
* feat(hitl): add human-in-the-loop approval for high-risk tool calls

* fix(hitl): address approval review findings

* docs(hitl): keep agent guidance under size limit

* fix(hitl): guard approval pause recovery

* docs(hitl): shrink inherited guidance

* fix(hitl): harden approval admission and recovery
2026-11-09 09:52:35 +08:00
Eilen Shin
aaca3eeaeb
refactor(agent): ground runtime decisions in structured evidence (#7986)
* refactor(agent): ground runtime decisions in structured evidence

Use producer status and structured errors for tool recovery, preserve memory context without lexical confirmation, and remove predetermined receipt and skill conclusions.

Require explicit MCP discovery and authenticated schedule trial actions. Preserve authorization, ownership, isolation, and bounded execution; update documentation and regression coverage.

* test(sandbox): expect read_file initialization error callback

* fix(agent): address evidence review follow-ups

* fix(agent): preserve retry budgets for OpenAI transport errors

* fix(agent): expose tool execution status in model requests

Project validated execution metadata and exact tool text into request-only JSON so status survives provider serialization. Preserve stored results and receipt hashes, cover SDK and graph boundaries, and migrate replay input hashes without changing recorded responses.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-08 22:00:23 +08:00
Eilen Shin
2a8927c5e1
feat(models): add StepFun provider preset (#7988)
* feat(models): add StepFun provider preset

* fix(models): standardize the StepFun preset label

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-08 09:49:43 +08:00
Wenchao An
c1f9064e9b
fix(models): route managed OpenAI GPT-6 presets through Responses (#7870)
* fix(models): route managed OpenAI GPT-6 presets through Responses

* test(memory): pin capacity scenarios to a stable clock

* fix(models): honor Flash mode for managed OpenAI GPT-6

* fix(models): reconcile preset effort with explicit reasoning contracts

* test(memory): share the fixed clock with updater capacity tests

* fix(models): normalize GPT-6 reasoning overrides and sampling

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-05 14:30:42 +08:00
Wenchao An
2145c9ab1f
feat(models): discover provider catalogs and search chat models (#7871)
* feat(models): discover provider catalogs and search chat models

* docs(models): scope catalog guidance to owning directories

* test(guidance): register model catalog scoped instructions

* test(memory): pin capacity scenarios to a stable clock

* fix(models): guard catalog imports and align picker interactions

* test(memory): share the fixed clock with updater capacity tests

* fix(models): clear provider filters removed by catalog refresh

* refactor(models): share connection fields across model forms

* fix(models): keep active provider filters clearable

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-05 06:43:01 +08:00
ZhiLin Zhang
e5969fd8b9
feat(harness): introduce HarnessGuardMiddleware for pre-write syntax validation and anti-doom-loop (#7624)
* feat(harness): introduce HarnessGuardMiddleware for pre-write syntax validation and anti-doom-loop

- Add HarnessGuardMiddleware to intercept tool writes and validate .py (AST) and .json syntax before disk write
- Add anti-doom-loop state machine detecting repeated error signatures (>=2) and A -> B -> A -> B oscillating loops
- Inject targeted remediation hints directly into tool results
- Add DynamicSkillRouter for zero-LLM JIT skill intent activation (<1ms)
- Fully unit-tested with 25 passing test cases

* fix(harness): address review findings on write tools, scoping, and content handling

* fix(harness): offload async validation off-thread, support Command results, and harden parsing

* fix(harness): address reviewer 1 feedback on scoping, assembly count, and non-blocking verification

- Revert .gitignore to exclude out-of-scope graphify-out entry
- Short-circuit syntax validation when block_on_syntax_error is false
- Trim AGENTS.md rule chain to fit hard guidance budget (AG002)
- Fix input_sanitization mock and update subagent middleware count expectation to 23

* fix(harness): preserve chunked write flow, normalize CRLF line endings, and guard error classification

- Preserve append-in-chunks flow: use codeop to distinguish incomplete Python fragments from syntax errors, and handle premature EOF on chunked JSON arrays/objects
- Reuse native _match_line_endings for str_replace to prevent CRLF line ending bypass
- Check existing tool failure status before classifying error text to avoid misclassifying successful tool outputs
- Add 4 regression tests covering chunked writes, CRLF replacement, and successful error-text reads

* fix(harness): scope pre-write syntax validation to str_replace and preserve write_file semantics

* fix(harness): make str_replace syntax check opt-in, allow progressive repair of invalid files, and clean up RFC docs

* chore: align subagent middleware count and satisfy AG002 guidance budget

---------

Co-authored-by: Yakyou7 <Yakyou7@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-04 20:13:19 +08:00
spud
21175ba46f
fix(runtime): persistently fence stale run event producers (#7619)
* fix(runtime): fence revoked run event producers

* fix(runtime): reduce off-mode producer guard locking

* docs: keep inherited guidance within size budget

* chore(runtime): remove development phase labels

* fix(persistence): normalize libpq SSL options for asyncpg connections

* docs(runtime): clarify producer fencing behavior in off mode

* test(persistence): align migration TLS probes with channel-binding rejection

* fix(runtime): authorize fenced Gateway history seeding

* fix(scheduler): preserve takeover identity in recovery publication

* fix(scheduler): report unconfirmed recovery publication

* fix(runtime): preserve history messages before the first human turn

* fix(runtime): clarify run creation failure diagnostic

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-02 15:26:42 +08:00
Hyeonsang Cho
17b05718e0
fix(agents): store an explicit null description as empty (#7638)
* fix(agents): store an explicit null description as empty

PUT /api/agents/{name} accepted "description": null and wrote it into the
agent's config, but AgentConfig.description is a str. Every later load of
that agent failed validation: the PUT itself, GET and repair PUTs returned
500, the file backend dropped the agent from the list, and the db backend
failed the owner's whole list.

Store an explicit null as the empty default, and have parse_agent_config
read a stored null description (already written by this route, or a bare
YAML key) as empty so affected agents load again.

* docs(changelog): reference #7638 for the agent null description fix

* docs(agents): document that a null description resets it to empty

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-01 23:34:59 +08:00
derrickLzw
51f0e2de1c
feat(extensions): scope configured middlewares to lead or subagent (#7661)
* feat(extensions): scope configured middlewares to lead or subagent

* fix(helm): synchronize configured middleware config schema version

* fix(extensions): require explicit configured middleware scope
2026-10-31 08:46:16 +08:00
Totoro
8d11aeb97b
feat(subagents): configure thinking and reasoning effort for native workers (#7668)
* feat(subagents): add thinking_enabled and reasoning_effort config for sub-agents (#4875)

(cherry picked from commit b1083728b23504cb0f1d1a463f8db70c5ec1c5a7)

* fix: resolve CI failures for subagent thinking config PR

- Fix UnboundLocalError in task_tool.py: initialize parent_thinking_enabled
  and parent_reasoning_effort before the if runtime block
- Remove unused imports from test_subagent_thinking_config.py
  (SimpleNamespace, pytest, resolve_subagent_model_name)
- Fix import sorting in subagents_config.py and test file
- Add reasoning_effort: None to executor test assertion

(cherry picked from commit 187851e307b871937e48bb67ae1b34b8b69bf92d)

* fix: address review comments - remove parent inheritance, supports_thinking clamp, Literal type

- Remove parent_thinking_enabled/parent_reasoning_effort from metadata
  extraction in task_tool.py (backward compatible: subagents only think
  when explicitly configured)
- Add supports_thinking clamp in _create_agent to prevent cross-model
  mismatch errors
- Change SubagentConfig.reasoning_effort to Literal type
- Fix sandbox_state assignment that was accidentally removed
- Fix syntax errors in subagents_config.py (missing closing parens)
- Update all docstrings/comments from "inherit" to "no override"
- Simplify test helper and remove parent inheritance tests

(cherry picked from commit 74f9c659ef4a98bd1756109929ab0748fab82317)

* feat(subagents): adapt reasoning settings to current native workers

* fix(subagents): retain model context on reasoning rejection

* fix(subagents): preserve model-factory reasoning diagnostics

---------

Co-authored-by: Neon549 <neon549@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-10-29 22:26:12 +08:00
huidongchen148
612140f183
feat(models): add managed Kimi K3.1 Pro capabilities (#7566)
Declare required thinking, native reasoning effort and image support for both
K3.1 Pro IDs. Normalize fixed parameters at the final Moonshot request boundary
and cover real SDK serialization with offline regression tests.

Fixes #7425
2026-10-27 14:57:29 +08:00
Tsai Yuan
89acf51caa
fix(tool-output): preserve prompt prefixes by default (#7455)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-27 10:09:03 +08:00
Totoro
ab8cd4e10f
fix(channels): keep IM binding codes on screen and report goal state in replies (#7591)
* fix(channels): keep IM binding codes on screen and report goal state in replies

Connecting Slack, Discord, Feishu, DingTalk, WeCom, Buzz or QQ now opens a
dialog that keeps the /connect command on screen with a copy button, its
expiry time and the binding status, instead of a toast that disappears.

The bots' /connect replies and the IM /goal replies now use the owner's
web UI language (else channel_connections.notification_locale), and /goal
reports the goal's state as the web goal bar does: in progress, why it
stopped, or that it was met.

* docs(changelog): note the IM binding dialog and localized IM replies (#7591)

* test(channels): check IM reply tables against every notification locale

The connect-reply check compared a fixed en-US/zh-CN pair, and the /goal
reply table had none. Both now match NOTIFICATION_LOCALES with the same
keys in each, like the existing _GOAL_REASON_TEXT check, so a new locale
fails here instead of raising KeyError at reply time.

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-10-27 09:46:58 +08:00
spud
aabfdeebc9
fix(streaming): make Redis stream END publication idempotent (#7572)
* fix(streaming): make Redis stream END publication idempotent

* fix(streaming): align Redis test fake with Lua END checks

Remove the unreachable owned missing-stream branch, mirror tail-only data checks in both fake script paths, and pin legacy END and owned missing-buffer behavior against real Redis.

* fix(streaming): gate legacy END scans after writer rollout

* fix(helm): synchronize embedded config version with schema

* test(runtime): use real stream bridge config in close coverage

Keep new configuration defaults in lifecycle fixtures and surface owner-task failures instead of masking them as event timeouts. Drain every helper task and cover context entry/close failures.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-27 00:16:53 +08:00
Hyeonsang Cho
ddfa45eaaf
fix(gateway): read the route authorization config off the event loop (#7557)
* fix(gateway): read the route authorization config off the event loop

AuthMiddleware resolves route permissions for every authenticated
request, and resolve_route_permissions read the hot-reloaded
authorization config inline. get_app_config() stats and hashes
config.yaml and extensions_config.json on every call and reparses them
after an edit, so that file IO ran on the event loop per request.

Read it through asyncio.to_thread, like the plugin authorization path
already does. The context copy keeps runtime config overrides visible.

* docs(changelog): reference #7557 for the off-loop route authorization config read

* docs(config): note the get_app_config read cost where it is documented

Move the off-loop note from gateway/AGENTS.md, which is 12 bytes under
its hard limit, to the Config Caching section that documents what each
get_app_config() read costs.
2026-10-26 18:44:09 +08:00
llb555
95ffb29e50
fix(config): keep atomic-write cleanup from replacing the caller's outcome (#7279)
* fix(config): keep atomic-write cleanup from replacing the caller's outcome

* docs(config): state the cleanup invariant for every exit path
2026-10-25 19:23:44 +08:00
haochen
f7f9a7971b
fix(sandbox): normalize mount container_path trailing slashes at config load (#7393)
* fix(sandbox): normalize mount container_path trailing slashes at config load

A container_path declared with a trailing slash (e.g. /mnt/reports/) was
accepted by VolumeMountConfig and normalized by LocalSandboxProvider when
building path mappings, but the sandbox tool path gates compare the raw
configured value, so every ls/read/write/glob/grep/bash access to the
mount failed with a confusing PermissionError. Normalize trailing
slashes in the config model so both layers agree, and reject the
container root '/' which the gates cannot express.

* fix(sandbox): normalize skills.container_path and pin empty-value rejection

Review follow-up on #7393:
- Share the container-path normalizer with SkillsConfig, whose prefix
  feeds the same as-is comparison in _is_skills_path and had the same
  trailing-slash dead-path failure mode.
- Pin the empty-string rejection: on main an empty container_path made
  the gates match every absolute path, applying one mount's read_only
  flag to all paths.
- Cover both config fields in CONFIGURATION.md.

---------

Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com>
2026-10-25 11:44:36 +08:00
Aling33
cb6ee72132
fix(config): delete read-only thread directories on Windows (#7446)
`Paths.delete_thread_dir()` called `shutil.rmtree()` directly. Windows
refuses to unlink a file that carries the read-only attribute
(`WinError 5`) where POSIX only needs a writable parent, so one read-only
file under `user-data/` made the whole thread undeletable: `DELETE
/api/threads/{id}` answered 500 `Failed to delete local thread data`,
`rmtree` had already removed everything it walked before that entry, and
every later retry failed on the same file.

Retry at the call site: clear the read-only attribute tree-wide and call
`rmtree()` once more. Only `PermissionError` is retried, the retry still
raises when it cannot help, so a file another process holds open stays a
real error instead of becoming a silent skip. A `onexc` handler cannot do
this portably: POSIX `_rmtree_safe_fd()` catches the handler's own
`PermissionError`, re-dispatches it as a `scandir` error and leaves the
deletion to fail later as `ENOTEMPTY`.

Co-authored-by: Aling33 <along308@users.noreply.github.com>
2026-10-25 11:02:49 +08:00
DDT
afe244e7f9
fix(config): pin explicit app config path across get_app_config (#7092)
* fix(config): pin explicit app config path across get_app_config

Fixes #7086. reload_app_config(config_path) now records an
_app_config_source pin (mirroring _extensions_config_source) so
get_app_config and DeerFlowClient(config_path=...) keep following the
explicit file instead of ambient default resolution.

* fix(config): synchronize app config cache and absolute pins

Serialize singleton reads, reloads, resets and injections so a concurrent call cannot publish a config under another file's pin. Keep explicit paths absolute across working-directory changes while preserving ambient discovery and failed-reload behavior.

Add deterministic concurrency and relative-path regressions. Validation: 33,875 offline tests and 262 blocking-I/O tests passed; frontend check, backend formatting and agent guidance passed.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-24 22:04:48 +08:00
Zeren Wang
74d2565f64
feat(projects): search shelf documents by keyword (#7286)
* feat(projects): search shelf documents by keyword

Add search_project_documents: two-tier name/summary then bounded content scan, case-insensitive substring matching with original-text offsets. Never converts on the fly; reports stop_reason, content_scanned, skipped reasons, and capped unscanned/truncated ids.

* fix(projects): derive unscanned count without paging after budget trip

Review feedback: once the content budget trips, stop paging list_active to enumerate unscanned ids — cap ids at the trip page and derive the honest total from one count_active query (tier 1 is exhaustive, so active - tier1 hits - processed = unscanned). Also add a tool-level isolation test pinning that search never crosses projects or users.

* fix(projects): scan the shelf over one stable snapshot, not offset pages

Review feedback: offset paging re-read the live shelf per page, so a concurrent upload/trash between pages could duplicate a match or silently skip one while still claiming complete coverage. Tier 1 now materializes the whole shelf in a single SELECT (one read snapshot) and tier 2 iterates those same rows; the budget-trip unscanned count is derived exactly from the snapshot with no extra query. Mid-scan upload/trash regression tests pin both failure modes.
2026-10-24 18:07:00 +08:00
Nan Gao
a9917e986b
docs(channels): align leader-lease failover and runtime-config notes with the merged code (#7420)
The two-Gateway harness checklist in CONFIGURATION.md said a crashed
leader hands its providers over within lease_seconds + grace_seconds; a
peer only tries to acquire on its fixed lease_seconds // 3 tick, so the
worst case is one renewal interval more (50 s with the defaults), as the
leader-lease section of the same file already says. The grace_seconds
field description had the same shorter bound.

The channels.leader_lease comment in config.example.yaml still said web
UI credential changes reach the holder only when it next starts the
channel; since #6988 the holder applies them within one renewal interval.

The harness `check` footer still listed the IM channel gates as waiting
for leader election, which landed in #6843; it now points at the manual
channel item in CONFIGURATION.md, which also gains the runtime-config step.
2026-10-24 16:35:26 +08:00
Totoro
0baa99ca90
feat(models): add scoped model controls to Web chats (#7249)
* test: stabilize reporter evidence and guard missing chat nodes

* feat(models): scope Web model choices to tasks, turns and conversations

* test(e2e): mock model-control preview for ordinary chat submissions

* fix(models): harden scoped choices and stale Agent submissions

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-10-22 08:08:23 +08:00
Hyeonsang Cho
d649cc73d2
fix(skills): create a missing extensions_config.json in the project root (#7048)
* fix(skills): create a missing extensions_config.json in the project root

When no extensions config resolved, the skill toggle created one at
Path.cwd().parent while the MCP writers used project_root(). Unless the
Gateway runs from backend/ with the project root at its parent, the
toggle's file landed outside the resolver's search path: the toggle was
lost on reload and the next MCP edit created a second file.

Add ExtensionsConfig.default_config_path() (the project root, the first
location resolve_config_path searches) and use it from the skill toggle
and both MCP create paths, so a created file is always the one the next
resolution reads back.

* docs(changelog): reference #7048 for the extensions config create path

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-21 22:43:49 +08:00
trent87
0d73798beb
feature(skills): keep allowed-tools as compatibility metadata (#6935)
* refactor(skills): keep allowed-tools as compatibility metadata

* docs(skills): align current guides with agent tool authority

* test(skills): simplify authority regressions and sync Korean docs
2026-10-20 19:34:28 +08:00
lakers-abing
f23a411325
fix(storage): resolve single-module colon backend imports (#7017)
* fix(storage): resolve single-module colon backend imports

* docs(storage): describe both backend import separators

---------

Co-authored-by: lakers-abing <236384465+lakers-abing@users.noreply.github.com>
2026-10-20 09:35:56 +08:00
RongJie G
bf2cb0e1df
fix(runtime): recover terminal Redis streams after producer loss (#7093)
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-19 15:53:54 +08:00
Jay0130-a
46c0899e53
fix(config): accept a UTF-8 BOM in operator-authored env and secret files (#7085) (#7112) 2026-10-19 15:48:10 +08:00
yangfucheng123
3aa144a377
fix(managed-models): map DeepSeek "minimal" effort to "low" (#6717)
The official DeepSeek Chat Completions endpoint rejects the generic
"minimal" effort token (issue #4514). The declared effort contract for
DeepSeek models now advertises the provider's real vocabulary
(low/medium/high/max/xhigh), defaults to "low", and aliases the generic
"minimal" preset onto "low" so the "flash" dialogue mode serializes a
value the endpoint accepts.

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-19 15:45:56 +08:00
Jay0130-a
dfb32ef52b
fix(agents): strip a leading BOM from SOUL.md before prompt injection (#7061)
Co-authored-by: Jay0130-a <Jay0130-a@users.noreply.github.com>
2026-10-18 21:51:39 +08:00
haotianqi666
07f0d74f70
fix(agents): clear incompatible knowledge defaults on self-update (#6995)
* fix(agents): clear incompatible knowledge defaults on self-update

* fix(agents): share knowledge default cleanup with HTTP updates

---------

Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com>
2026-10-18 11:12:57 +08:00
Totoro
16d36ccd21
feat(sandbox): add bounded AIO prewarming and lifecycle benchmarks (#6998)
* fix(benchmark): validate sandbox lifecycle and concurrent measurements

* feat(sandbox): prewarm likely AIO tool turns

(cherry picked from commit 3a1017ed032ed9331987bc3dc61f23dcdac4e905)

* fix(sandbox): bind bounded prewarming to execution leases

* docs: trim frontend guidance below its existing size limit

* fix(benchmark): separate failed lifecycle stage timings

* test(sandbox): measure prewarm overlap and unused resources through agent graphs

* fix(sandbox): finish lease and sampler cleanup after cancellation or errors

* docs(benchmark): distinguish initialization wait from model overlap

* docs(helm): align the configuration example with version 61

* fix(sandbox): preserve synchronous agent graphs when prewarming is enabled

* test(cargo): isolate native fixtures from inherited make jobserver flags

* fix(sandbox): require managed cleanup before prewarming

* fix(sandbox): log denied prewarming as an expected skip

* fix(benchmark): filter provider summary input records

---------

Co-authored-by: nothingyuancando <1579035512@qq.com>
2026-10-18 08:10:56 +08:00
Totoro
2464566942
fix(summarization): cover all selected history before compaction (#6918)
* fix(summarization): cover all selected history before compaction

* fix(summarization): bound compaction work and recover oversized carries

* docs: leave headroom in middleware guidance chain

* fix(summarization): cool down repeated chunk-limit failures

* test(runtime): complete callback cleanup graph fixture

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-16 09:41:20 +08:00
Nan Gao
e8c7df9042
feat(channels): arbitrate IM transports with a per-provider leader lease (#6843)
* feat(channels): arbitrate IM transports with a per-provider leader lease

Every Gateway instance connected every enabled IM platform with the same
bot credentials, so channels could not run on more than one instance
(Telegram 409 storms, Discord answering twice, scattered per-process
state). channels.leader_lease was reserved by #6746; this implements it.

- channel_leases table (migration 0041): one row per provider. Acquire is
  one dialect-native upsert decided against the database clock and bumps
  a fencing epoch; renew extends only a live lease at the holder's epoch;
  release sets the expiry to 0 so a peer takes over on its next tick.
- app/channels/leadership.py: the holder's local monotonic deadline
  (send time of the last confirmed renewal + lease_seconds); holds() turns
  false one renewal interval early, so the transport stops at least
  grace_seconds before a peer may connect. DB calls are bounded.
- ChannelService: start fence in _start_channel (no DB call) behind every
  start path; fixed-rate supervisor renews/acquires and runs transport
  starts/stops as per-provider tasks; takeover reloads the provider's
  saved config; failed starts back off; shutdown releases only leases
  whose transport stopped. get_status reports leader/leader_here and the
  holder's transport state on every instance.
- Notification worker leaves rows of providers another instance holds.
- Gate: the enabled=false refusal now recommends auto/true; an explicit
  true with database.backend memory is refused when a channel is enabled.
- Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated.

* docs(changelog): link the channel leader lease entry to #6843

* fix(channels): see peer-held leases for providers absent from local config

Review on #6843: a provider connected from the web UI on a peer never
enters this instance's channel config, so the notification worker here
still claimed its rows (and parked them for 900s) and GET /api/channels/
reported no leader for it. Every supervision round, and startup, now also
reads the whole lease table once (bounded), and notification exclusions
and status use those observations: a provider whose live lease a peer
holds is left to it even without local config.

Running that read concurrently with renewals exposed that, through
SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on
another connection can hand back one of its own bind values instead of
the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do
not show it). A lease must never take such a row for a confirmed
renewal, so acquire/renew/release now decide by affected-row count and
acquire reads its epoch back in the same transaction.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-14 13:51:42 +08:00
OUAO-FRANK
12397d2142
fix(sandbox): forward bash command timeout to remote providers (#6782) 2026-10-14 06:47:38 +08:00
Eilen Shin
e98179a9d3
feat(models): register managed models with provider presets (#6757)
* feat(models): register managed models with provider presets

* docs: add PR 6757 UI comparison screenshots

* test: fix model management CI checks

* docs: remove PR 6757 screenshots

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-13 08:14:23 +08:00
Nan Gao
d18d2937e7
feat(channels): add channels.leader_lease and gate an explicitly disabled lease under multi-process signals (#6746)
* feat(channels): add channels.leader_lease and gate it under multi-process signals

IM channels other than GitHub connect outbound to their platform. With more
than one Gateway instance each one opens that connection with the same bot
credentials: Telegram rejects the second poller with 409, Discord answers every
message twice and per-process ChannelManager state diverges. The multi-process
startup gate said nothing about channels; the only guard was a comment in the
Helm values.

Add the `channels.leader_lease` section (`enabled: auto|true|false`,
`lease_seconds`, `grace_seconds`; startup-only, registered in the reload
boundary) that the per-provider leader lease will read, validated by a Pydantic
model from the non-schema `channels` mapping. The gate now refuses
`enabled: false` under a multi-process signal while a non-GitHub provider is
enabled under `channels.*` or connectable through `channel_connections`, naming
the providers and the rollback; a malformed block is refused on every topology.
`auto` follows the multi-process signal, so a single instance is unchanged.

ChannelService strips the block before its per-provider loops and exposes it
as `leader_lease`; `ensure_ready_channels` only visits registered providers.
The two-Gateway harness overlay leaves the block alone. config_version 60 in
the example, the Helm chart and its README; docs replace the comment-only
"keep 1 replica" guidance with the gate.

* docs(changelog): link #6746

* fix(channels): stop recommending the unimplemented leader lease as a remedy

Review on #6746: the refusal told operators to switch channels.leader_lease
from false to auto/true "so one instance owns each provider's connection", but
the transport arbitration ships separately, so that change only bypassed the
clause and every worker still started the same bot.

The refusal now offers only the remedies that work today (a single instance, or
disabling the channels) and states that auto/true are reserved for the lease
that ships separately. A regression helper in the gate tests asserts the
remedies start with the rollback and that auto/true are never offered. The
config example, Helm values/README, CONFIGURATION.md, README, the deployment
guide (en/zh), the model docstrings and both changelogs describe auto/true as
reserved settings rather than an ownership guarantee.

* fix(channels): warn on an enabled unknown channel key; reword the changelog entry

Review on #6746: the registry guard in ensure_ready_channels skipped a
misspelled provider key (channels.telegam) silently, where it previously
reached _start_channel and logged 'Unknown channel type'. An enabled dict
block that is neither a registered provider nor leader_lease now logs a
warning that lists the registry.

The changelog entry claimed the chart and docs drop the "keep 1 replica"
guidance; they keep it (auto/true do not make several replicas safe yet) and
only stop relying on the comment alone.

* fix(dev): disable UI channel providers in multi-instance overlay

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-12 23:05:16 +08:00
Hyeonsang Cho
29a958100e
fix(sandbox): stop resolving sandbox and ACP env values a second time (#6673)
* fix(sandbox): stop resolving sandbox and ACP env values a second time

AppConfig.resolve_env_variables already replaces every $VAR when
config.yaml loads. The AIO, E2B and OpenSandbox providers and the ACP
tool then resolved the result again, so a secret that itself starts with
$ (a bcrypt hash, a generated password) was looked up as a variable name
and reached the sandbox or agent as "" -- or as an unrelated host
variable's value when that name existed.

Pass the loaded values through unchanged, as BoxLite and the local
provider already do. The removed helpers' str() coercion was dead: both
fields are typed dict[str, str] and reject non-strings at load.

* docs(changelog): reference #6673

* docs(config): pin when sandbox and ACP env values are resolved

ACPAgentConfig.env and the SandboxConfig docstring said $ values are
resolved from the host environment without saying when, which reads as
"at spawn time" and invites a consumer-side second pass again. Both now
say it happens when config is loaded, matching SandboxConfig.environment.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-11 17:50:04 +08:00
Nan Gao
db7340d2fa
fix(security): add DEER_FLOW_CREDENTIALS_KEY and wire the channel credentials cipher (#6611)
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption

There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.

- New harness module deerflow.config.credentials_key: env-only
  DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
  all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
  decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
  values still readable, and get_credentials_cipher() as the process-wide
  entry point. Malformed keys are refused without echoing key material.
  Without the variable a single instance generates
  {base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
  create (0600) plus read-back with a bounded settle window, shared by
  .credentials_key, .jwt_secret (fixes two replicas cold-starting on a
  shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
  constructors); the Gateway loads the cipher off the event loop at
  startup when channel_connections is enabled and passes it to every
  ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
  multi-instance deployment with channel_connections enabled and no key
  is refused. Plain multi-worker shares the generated file and is allowed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh

- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
  the urlsafe alphabet) and preserves it across upgrades via lookup like
  AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
  so a user-managed existingAppSecret without it still boots (the Gateway
  gate refuses only multi-instance deployments that store credentials).
  README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
  generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
  noclobber/O_EXCL with read-back): the same file the Gateway generates in
  its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(security): make abandoned secret-file replacement single-winner

A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(security): name the setting that turns off a credentials-key consumer

The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(security): never expose a secret file before its value is written

O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(models): recover an empty managed-model key when no catalog exists

An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start

scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:51:09 +08:00
Zeren Wang
553f85e788
feat(projects): best-effort LLM summaries for shelf documents (#6581)
Opt-in per-document summary column rendered into the injected
<documents> index, list_project_documents JSON, and the documents API.

Pipeline (projects/summaries.py, behind the narrow enqueue_summary
interface):
- The two shelf write routes enqueue a job on created=True; dedup hits
  never regenerate. Off by default (projects.summaries_enabled).
- Bounded worker pool (summary_concurrency + bounded summary_queue_size;
  a full queue drops the job) fed by FIFO; shutdown drains queued jobs,
  delivers one sentinel per worker via blocking put (queue smaller than
  the worker pool cannot strand idle workers), and bounds only
  genuinely in-flight jobs with the grace period.
- Jobs carry (document_id, project_id, user_id) explicitly; the job's
  owner is bound into the user ContextVar around the model call so
  tracing/usage attribution never falls back to 'default'.
- Text source reuses read_text_serving_path (convertible-first rule,
  content validation); conversion has no deadline (await_drained), the
  model call has a caller-side wait limit — the worker slot is retained
  until the provider call actually finishes and late results are
  discarded.
- Output contract: strip leading <think> reasoning -> unwrap fence/
  quotes -> collapse to one line -> empty is NULL -> UTF-8 byte cap on
  a character boundary; length-capped responses (provider-normalized
  detectors) are failures.
- set_summary pins updated_at (literal self-assignment) so a landing
  summary never reorders the shelf; owner-predicated UPDATE is a no-op
  on rows purged mid-generation.
- Metrics: eligible/success/failed/skipped/queue_dropped/
  uncapped_accepted.

Schema/config:
- project_documents.summary (migration 0036, nullable, no backfill;
  forward-compat floor unchanged).
- Six projects.summary_* keys, all registered in STARTUP_ONLY_FIELDS
  (captured once at lifespan startup); config_version 57 -> 58.
- run_oneshot_llm gains an optional model_overrides passthrough (passed
  only when set, existing callers unchanged) and a detailed variant
  exposing the raw response message for termination-signal detection.

Docs: API.md response field, ARCHITECTURE.md pipeline paragraph,
migrations AGENTS.md chain head, root AGENTS.md records the scoped
all-core exception with the lift-out path.
2026-10-10 07:11:02 +08:00