93 Commits

Author SHA1 Message Date
Nan Gao
5872d07e9a
fix(gateway): gate declared multi-instance deployments and require the redis stream bridge (#6328)
* fix(gateway): gate declared multi-instance deployments and require the redis stream bridge

GATEWAY_WORKERS / WEB_CONCURRENCY only count the uvicorn workers of one
process tree, so a Kubernetes Deployment with replicas > 1 (one worker
per Pod) passed every multi-worker startup check while each Pod's
startup orphan reconciliation wrote the other Pods' lease-less runs off
as crashed on every rolling update.

Add `deployment.multi_instance` as an explicit declaration that runs the
same gate, plus a DEER_FLOW_MULTI_INSTANCE env override so deploy tooling
can set the declaration from its replica count without editing
config.yaml (wiring the Helm chart to export it is a separate change).
Both the worker count and the declaration now also require a Redis
stream bridge and refuse an explicit memory sandbox ownership store,
since the memory bridge and ownership store are process-local. The
agent-storage divergence warning, the inbound webhook dedupe warning and
the WeChat QR-login guard honor the same declaration.

* docs(changelog): link the multi-instance gate entry to #6328

* fix(ci): bump chart config_version to 53 and keep AGENTS.md under the size limits

config.example.yaml moved to config_version 53 for the new deployment
section, so the Helm chart's embedded config and its README example
follow. The multi-instance notes in backend/AGENTS.md and
backend/app/gateway/AGENTS.md shrink to one line each: main already sits
within 55 bytes of the per-file hard limit for the gateway guide and
within 163 bytes of the chain hard limit under backend/, and the full
contract lives in backend/docs/CONFIGURATION.md.

* fix(gateway): name every active knob in the multi-process rollback

When GATEWAY_WORKERS > 1 and a multi-instance declaration are both
active, the refusal offered only "Set GATEWAY_WORKERS=1", so an operator
who followed it bounced through a second refusal on
DEER_FLOW_MULTI_INSTANCE / deployment.multi_instance at the next start.
The rollback now withdraws both knobs, and the browser-tools refusal
takes the same path instead of the worker-count-only message.

multi_instance_declaration() returns a frozen MultiInstanceDeclaration
(source, value) with knob/rollback properties instead of a display
string, so deps.py no longer branches on message prose. Tests pin the
combined rollback text and that following it actually clears the gate.

* docs(gateway): keep the gateway AGENTS.md under the size hard limit after merging main

main grew backend/app/gateway/AGENTS.md by 30 bytes, which left it 25 bytes
under the 49152-byte hard limit; the one-sentence multi-instance note from
this branch pushed the merged file to 49164. Shorten that sentence by 18
bytes so scripts/check_agent_guidance.py passes on the merge result.

* ci: re-run checks after test_stop_cancels_a_hung_driver_poll flaked on shard 4

tests/test_mcp_task_service.py::test_stop_cancels_a_hung_driver_poll timed
out once on backend-unit-tests (4) for 2c16206ac. The MCP task service and
its test are identical to main, main's run for cfe2023df passed, and 15
consecutive local runs pass; the test gives service.stop() a 1s budget
while cancellation drain allows 5s, so a loaded runner can exceed it.
Fork contributors cannot re-run a job, hence this empty commit.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-05 22:34:57 +08:00
Totoro
cdbcc0d9e9
feat(scheduled-tasks): show goal outcomes and end conditions in run history (#6326)
* docs(scheduler): document tool_enabled, goal notices and goal-evaluation change

Follow-up to #6229:
- CONFIGURATION.md: add scheduler.tool_enabled to the example and notes.
- CONFIGURATION.md, README (en/zh/pt): unmet goal runs send a goal-unmet
  notice instead of a completion notice, and three unmet runs add an
  auto-pause notice; the docs still said only success/failed notify.
- CHANGELOG (en/zh): the #6229 entry now states the default-on goal
  evaluation change for scheduled/webhook/autonomous goal runs and that
  evaluator usage counts toward run usage.

* feat(scheduled-tasks): show goal outcomes and end conditions in run history

Follow-up to #6229. The API already returns these fields; the tasks page
did not show them, so a met goal read as plain "Success", an unmet
reason code rendered as a red execution error, and the goal-unmet IM
notice pointed to a workspace showing neither objective nor verdict.

- Task detail shows the per-run goal, max automatic runs and end time
  when a conversation-created task has them.
- Goal runs show "Goal met" (or met relying on stated assumptions);
  unmet runs show a localized reason plus the host code in neutral
  styling. The run whose agent asked to stop the schedule is marked.
- Runs without a goal, and failed runs, render as before.

Read-only and frontend-only: no API change.

* docs(changelog): add #6326 scheduled-task goal outcome entry

* fix(scheduled-tasks): show readable unmet reasons instead of raw codes

Unmet runs now show only the localized reason; the host code moves to the
tooltip so it can still be matched with the IM notice. Codes without a
label are still shown verbatim.

* fix(scheduler): show readable goal reasons instead of internal codes

Task detail now localizes the host-written last_error values from goal and
stop finalization (agent stop, automatic pause, unmet reason codes) under a
matching label, and Chinese copy is tightened. Goal-unmet and auto-pause IM
notices state the reason in words instead of printing the raw code; unknown
codes are still not forwarded.

* docs(changelog): describe readable goal reasons in #6326 entry

* test(scheduler): pin host goal strings with a cross-language contract

Address review on #6326:
- Add contracts/scheduled_goal_notes_contract.json listing the task
  last_error strings and unmet reason codes the tasks page and IM notices
  translate. A backend test checks finalization constants, blocker and
  stand-down codes against it; a frontend test checks every listed value
  has a label.
- Move the two last_error strings in finalization.py into constants.
- Frontend reason lookup strips the blocked: prefix like the IM map and
  now covers goal_not_met_yet, with en/zh labels.

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-05 21:56:04 +08:00
Grapette.L
e7e5a5e9a3
fix(gateway): read uvicorn's WEB_CONCURRENCY in the remaining worker-count readers (#6253)
* fix(gateway): read uvicorn's WEB_CONCURRENCY in the remaining worker-count readers

* fix(gateway): name the worker spelling an operator can act on

The browser refusal and the dedupe warning both took the count but printed
GATEWAY_WORKERS, so on the launches that pass uvicorn no --workers the advice
silences the gate while the extra processes keep starting. Resolve which
variable set the count and print that, and let an unparsable GATEWAY_WORKERS
fall through to WEB_CONCURRENCY instead of reporting a single worker.

Also stop the comments from claiming the readers are equivalent: they share
the two spellings and the blank-means-unset rule, but channel_connections.py
turns an unparsable value into 0 and refuses while these stay inert.

* docs: state the single-worker requirement under both spellings

The browser tool group and the inbound dedupe warning now refuse or warn on
WEB_CONCURRENCY as well, so guidance that names only GATEWAY_WORKERS=1 is
incomplete for the shipped Dockerfile and serve.sh launches, where uvicorn
takes the process count from WEB_CONCURRENCY.

* test(features): isolate the browser capability surface from the shell's worker count

browser_capability() asks for the Gateway worker count, which this change makes
readable from WEB_CONCURRENCY, so an exported count in the invoking shell turned
test_features_reports_browser_control_enabled_when_configured_and_runtime_available
into a failure that has nothing to do with the router.
2026-10-04 16:36:01 +08:00
creed
9eb01b371a
feat(config): select an explicit backend dotenv file (#6227)
Signed-off-by: 97three <2212371308@qq.com>
2026-10-03 23:11:48 +08:00
creed
7bb3624c06
feat(serper): add configured source-domain filters (#6228)
* feat(serper): add configured source-domain filters

Signed-off-by: 97three <2212371308@qq.com>

* fix(helm): align embedded config version with example

Signed-off-by: 97three <2212371308@qq.com>

---------

Signed-off-by: 97three <2212371308@qq.com>
2026-10-03 21:58:50 +08:00
Hyeonsang Cho
7f7668d975
fix(community): refuse non-global addresses in the shared SSRF guard (#6202)
* fix(community): refuse non-global addresses in the shared SSRF guard

is_blocked_address ORed is_private, is_loopback, is_link_local,
is_reserved, is_multicast and is_unspecified. None of them covers the
100.64.0.0/10 shared address space, which holds CGNAT and Tailscale
hosts and Alibaba Cloud's 100.100.100.200 instance metadata endpoint, so
web_fetch, web_capture, the agentic browser and personal MCP connections
could reach it, including through the IPv4-mapped ::ffff:100.100.100.200
form a DNS answer can carry.

Also refuse any address that is not is_global, the classification the
repo's other SSRF filters already use. Keep the existing flags alongside
it: some non-public forms still report is_global, such as the NAT64
spelling of a metadata address (64:ff9b::a9fe:a9fe) and 224.0.0.1, so an
is_global-only check would newly allow them.

* docs(changelog): reference #6202 in the shared address space entry

* test(community): pin the IPv4-mapped metadata spelling at the web_fetch and personal MCP entry points

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-03 10:29:56 +08:00
Insecshin
759136e074
fix(config): correct Gemini official endpoint reasoning example (#6161)
* fix(config): correct Gemini official endpoint reasoning example

* docs: clarify official Gemini endpoint heading
2026-10-02 22:32:21 +08:00
creed
f4af75a7fd
feat(community): add bounded retries for Jina web fetch (#6143)
* feat(community): add bounded retries for Jina web fetch

Closes #6116

Keep retries opt-in and share a total deadline across requests and backoff. Preserve cancellation and the existing Error contract. Include offline DNS and local-wheel fixture corrections needed to run the complete backend suite in the isolated verifier.

Signed-off-by: 97three <2212371308@qq.com>

* docs(community): scope Jina retry guidance to its provider

Signed-off-by: 97three <2212371308@qq.com>

* fix(community): jitter bounded Jina retry backoff

* docs(community): apply provider-scoped Jina guidance

Signed-off-by: 97three <2212371308@qq.com>

---------

Signed-off-by: 97three <2212371308@qq.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-02 21:35:56 +08:00
creed
71f109275b
feat(serper): support relative search time ranges (#6113)
* feat(serper): support relative search time ranges

Signed-off-by: 97three <2212371308@qq.com>

* docs(serper): address review and trim inherited guidance

---------

Signed-off-by: 97three <2212371308@qq.com>
2026-10-01 20:12:43 +08:00
Totoro
b57fa6fb9e
feat(scheduler): push scheduled task results to IM channels via a notification outbox (#6135)
* feat(persistence): add notification delivery outbox for scheduled task IM push

New notification_deliveries table and repository (migration 0012) backing
the outbox pattern for issue #4254: idempotent enqueue keyed on task_run_id
plus event plus provider plus target, atomic claim of due rows, and
exponential-backoff retry state moving pending to sending to sent or failed.
Execution status stays in scheduled_task_runs; delivery status lives here.

* feat(scheduler): enqueue and deliver scheduled-task notifications over IM channels

Closes the loop for issue #4254. On run completion the scheduler writes
outbox rows for the task owner's connected channel identities (success and
failure events; interrupts stay silent), and a new NotificationDeliveryWorker
polls due rows and pushes them through the channel's proactive send path.
Channel gains send_notification; WeCom implements it via the WS client's
send_message with errcode checking, the base class raises for providers
without proactive push.

Delivery is best-effort and never shadows execution status: enqueue and send
failures are logged and recorded in the outbox, which retries with backoff.
Channel liveness is re-checked at delivery time, and the worker stops before
the channel service on shutdown. Feature activates only when channel
connections are enabled; no new dependencies.

Verified end to end against a live WeCom bot: scheduled run completion
produced an outbox row that was delivered in under one second.

* feat(scheduler): include scheduled run result summary in IM notifications

The v1 notification only carried the run outcome skeleton (status, task id,
run id). Completed runs now include the agent's final answer so the IM push
is actually useful headlessly.

- The delivery worker accepts an optional resolve_run_summary(run_id,
  user_id) resolver; for run_completed events it enriches the rendered text
  with a "Result:" line (bounded to 1000 chars). The lookup is best-effort:
  resolver failures or missing summaries degrade to the skeleton text and
  never block or fail the delivery.
- The summary is read at delivery time from runs.last_ai_message (the
  convenience column written on run completion), not snapshotted into the
  outbox payload; retried deliveries re-read the freshest value and the
  outbox row payload stays skeleton-only. The claimed outbox row is never
  mutated in place.
- Gateway wiring scopes the read to the outbox row's owner_user_id so a
  stale row cannot pull another user's run content.
- Failed runs keep the error-only skeleton: a partial answer on a failed
  run would be misleading.

Tests: 9 new worker cases (rendering incl. truncation, sync/async
resolvers, failure/missing fallbacks, no resolver call for failed runs).
Full scheduled + outbox + channels regression green (441 passed), ruff
check + format clean.

* fix(scheduler): address review feedback on notification delivery reliability

Review follow-ups for the scheduled-task IM notification outbox:

1. Crash recovery for "sending" rows: claim now stamps updated_at and the
   worker reconciles rows stuck in "sending" past a 10-minute lease back to
   "pending" before every claim. Per-row try/except isolates a crashing
   delivery from the rest of the claimed batch.
2. Double-send race under multiple workers: claim returns exactly the rows
   its own UPDATE...RETURNING flipped (guarded by status='pending'), so a
   rival worker's concurrent claim can no longer be handed out twice.
3. Restore non-ASCII comments and drop the BOM accidentally introduced in
   test_scheduled_task_service.py.
4. Manual "run now" completions no longer enqueue IM notifications; the
   user is already watching the UI. Missing trigger metadata fails safe
   towards notifying.
5. Channel-outage failures no longer consume the retry budget: they are
   parked with count_attempt=False on a flat 15-minute backoff so an
   hours-long outage cannot exhaust retries and drop the notification.

Tests: repository 16, worker 21, scheduled-task service 31; targeted
regression across notification/scheduler/channel suites: 511 passed.

* fix(scheduler): address round-2 review on IM notification delivery

Park WeCom transport/outage failures without burning retries, bound worker stop, include task titles, align enqueue with delivery, and document the IM push behavior.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(scheduler): complete IM notification docs and drop unused list API

Add backend/channels/migrations AGENTS notes, CHANGELOG and config.example hints for the outbox push path, and remove the unused list_by_task_run repository method flagged in review.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(scheduler): address round-3 review on IM notification delivery

Cap channel-down parking by row age and parked attempts, redact IM egress
content, narrow WeCom transport exceptions, add lifespan wiring tests, and
run ruff format so lint-backend passes.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(scheduler): redact entire PEM blocks in IM egress text

The previous BEGIN-only pattern left key material and the END marker in notifications.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(persistence): align 0014 parked_attempts with bootstrap schema tests

Backfill existing rows with a temporary server default, then drop it so create_all matches Alembic, and move HEAD anchors to 0014.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(persistence): format notification tests and shorten alembic revision id

ruff format --check failed on test_notification_delivery_worker.py.
0015_notification_delivery_parked_attempts exceeds alembic_version.version_num
(VARCHAR 32) and truncates on Postgres CI; use 0015_parked_attempts instead.

* test: fix post-merge CI failures after migration renumber

Update migration 0015 test HEAD expectation to 0017_parked_attempts and add queue_timeout_seconds to the lifespan notification worker mock scheduler config.

* fix(persistence): chain the notification outbox migrations after 0026

Main's head moved from 0016 to 0026_mcp_task_lease_tokens, so the two
outbox revisions are renumbered and re-parented:

- 0017_notification_deliveries -> 0027_notification_deliveries
- 0018_parked_attempts -> 0028_parked_attempts

Only the revision ids, parents and the migration guide change.

* fix(scheduler): adapt the notification hook and its docs to main

- tests: the completion tests assert on the atomic complete_run record;
  new cases cover a completion that was not recorded, a task deleted
  meanwhile, a failed task read, and the outbox being off.
- AGENTS.md: drop the root note and shorten the backend note to a pointer;
  the two notes pushed two effective guidance chains past the 96 KiB limit.
- CHANGELOG: move the entry from the released 2.1.0 section to Unreleased
  and add its link references.

* test(persistence): move the chain-head pin to the outbox migrations

test_migration_0026 pinned 0026 as the head, so it failed once the outbox
revisions chained after it. It now checks chain membership like the older
migration tests, and a new test file owns the head pin, checks the parents
and the 32-character revision id limit, and runs the upgrade and downgrade.

* fix(gateway): wire scheduled-task notifications after the channel service starts

The outbox wiring sat in the scheduler block and asked for the running
channel service there. #5035 moved the channel service start to after the
scheduler, and the two changes merged without a conflict, so on every
startup the lookup returned None and the feature disabled itself with the
"write-only outbox" warning. The lifespan tests patched the lookup to a
constant and kept passing.

The wiring now runs in its own step after the channel service start. The
scheduler is built as on main, and ScheduledTaskService gains
attach_notification_outbox() so the enqueue side is enabled once the
delivery worker is running. The guard, the worker and the shutdown order
are unchanged.

Tests: the lifespan helper can model the real accessor (None until the
channel service has started); a new case fails without this fix.

* fix(scheduler): enqueue from scheduler start and keep the push when the title read fails

Follow-up to the startup-order fix, from review.

- The enqueue side is wired with the scheduler again (it only needs the
  durable table), and only the delivery worker waits for the channel
  service. Attaching it later left a window after the scheduler started in
  which a finished run wrote no outbox row. If the channel service is
  missing or the worker cannot start, the enqueue side is detached, so the
  outbox still never becomes write-only.
- A failed task read after complete_run no longer drops the notification:
  the title is optional and the message falls back to the task id. A task
  deleted meanwhile still stays silent.

* fix(channels): park WeCom notifications while the SDK socket is closed

The WeCom SDK raises a plain RuntimeError when its socket is not open, and
send_notification only mapped ConnectionError, TimeoutError and OSError to
ChannelUnavailable. A disconnect therefore spent the counted retry budget
and settled the row as failed after about 15 minutes, although the docs
say transport outages park without consuming retries.

send_notification now asks the client's is_connected first. The new test
uses a real, never-connected SDK client: it raised RuntimeError after three
retries before, and parks now.

* docs(scheduler): describe which outcomes notify; mirror the changelog entry

- README (en/zh) and CONFIGURATION.md: only runs that finish as success or
  failed notify; occurrences that end without a finished run (launch error,
  queue timeout, restart recovery) do not. Parking lasts up to about a day.
- CHANGELOG: cite the issue as text like the rest of the file, and add the
  entry and link reference to CHANGELOG_zh.md.
- AGENTS.md: drop the backend pointer so this branch adds nothing to the
  guidance chains closest to their size limit; the scheduler-side rules
  and the lifespan order now live in app/channels/AGENTS.md.

* docs(changelog): cite #6135 next to #4843 for the notification outbox entry

* fix(scheduler,channels): re-check the IM binding at delivery time; park WeCom transport errors, count its rejections

Review round 1 on #6135.

The delivery worker now takes resolve_connections (the Gateway wires
ChannelConnectionRepository.list_connections) and, before the channel
liveness check and the summary read, requires the row's owner to still
hold a connected (provider, external_account_id). A disconnected target
is finalized with mark_failed(terminal=True); a lookup failure or an
unexpected result shape parks the row. Terminal outbox rows ignore
later status writes.

WeCom: the SDK raises RuntimeError from send_message for a socket that
is not open and for a dropped connection, but also for a platform
rejection ("Reply ack error: errcode=..."). Only that prefix keeps
counting; other RuntimeErrors, ConnectionError/TimeoutError/OSError and
websockets' ConnectionClosed park the row as ChannelUnavailable.

* fix(channels): treat a WeCom ACK rejection as proof the transport is up; pin the rejection prefix through the retry branch and the real connection-row shape

---------

Co-authored-by: GaoTianyuanhr <75788039+GaoTianyuanhr@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: GaoTianyuanhr <GaoTianyuanhr@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-10-01 17:59:30 +08:00
Hyeonsang Cho
ed09efdb7f
fix(config): upgrade the config.yaml the Gateway loads (#5991)
* fix(config): upgrade the config.yaml the Gateway loads

scripts/config-upgrade.sh resolved DEER_FLOW_CONFIG_PATH, then
backend/config.yaml, then <checkout>/config.yaml. make dev exports
DEER_FLOW_PROJECT_ROOT=<checkout>, so the Gateway reads the checkout copy
first: with both files present the script upgraded a file nothing loads
and left the live config outdated while reporting success. It also
ignored DEER_FLOW_PROJECT_ROOT and a DEER_FLOW_CONFIG_PATH set in .env,
and fell back to another file when DEER_FLOW_CONFIG_PATH named a
missing one.

Ask the harness resolver (AppConfig.resolve_config_path) instead of
copying its order, defaulting DEER_FLOW_PROJECT_ROOT to the checkout as
serve.sh does. A missing DEER_FLOW_CONFIG_PATH or invalid project root
now fails with the Gateway's error; no config anywhere still creates
<checkout>/config.yaml from the example.

* docs(changelog): reference #5991

* test(config): pin config-upgrade seeding a missing config from the example

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-29 22:10:29 +08:00
lekt8
f889ce7517
feat(community): add Unbrowse web fetch provider (#5981)
* feat(community): add Unbrowse web fetch provider

Add a community web_fetch provider backed by Unbrowse
(https://unbrowse.ai), a hosted service that returns a page as markdown,
over plain HTTP when that is enough and through a cloud browser when the
page needs JavaScript. The tool sends one JSON-RPC tools/call POST
(unbrowse.scrape) to Unbrowse's MCP endpoint, so it needs no extra Python
package and no MCP client (uses httpx, already a dependency).

Unbrowse is not a general web search engine, so only web_fetch is added.

Changes:
- backend/packages/harness/deerflow/community/unbrowse/{__init__,tools}.py
  web_fetch_tool with the same output shape as the Sofya, Exa and
  Firecrawl fetch tools (# <title> plus markdown, 4KB cap). API key is read
  from the config.yaml api_key field or the UNBROWSE_API_KEY env var.
  Optional render: auto|never|always; unknown values log a warning and fall
  back to auto. JSON-RPC errors, isError tool results, HTTP errors and
  malformed payloads come back as "Error: ..." strings.
- backend/tests/test_unbrowse_tools.py: key resolution, render handling,
  request shape, structuredContent vs text payloads, error paths.
- scripts/doctor.py, scripts/wizard/providers.py: register the key and the
  web_fetch provider; backend/tests/test_doctor.py covers both doctor paths.
- config.example.yaml, .env.example, backend/docs/CONFIGURATION.md,
  tools/AGENTS.md, EN/ZH harness/tools.mdx: document the provider.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(changelog): add Unbrowse web_fetch provider entry

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-28 15:08:49 +08:00
lihongyuan99
33a423d375
docs(backend): drop references to APIs that do not exist (#5798)
* docs(backend): drop references to APIs that do not exist

PATH_EXAMPLES.md's upload example imports THREAD_DATA_BASE_DIR from
thread_data_middleware; the module exists but exports no such name, so the
snippet fails with ImportError as written. Resolve the same directory through
the accessor the middleware itself uses, get_paths().sandbox_uploads_dir(),
which yields the layout the file's own path table documents.

CONFIGURATION.md's Docker sandbox block lists `auto_start: true`. SandboxConfig
declares use/port/container_prefix/mounts/network but not auto_start, and the
provider is constructed as cls(**kwargs) from the caller of
get_sandbox_provider() rather than from config extras, so the key is accepted by
extra="allow" and then silently dropped. Remove the line from the example.

* docs(backend): resolve the uploads example through the owner-scoped bucket

get_paths().sandbox_uploads_dir(thread_id) without a user_id selects the legacy
threads/<id>/ bucket, but Gateway uploads land in
users/<user_id>/threads/<id>/user-data/uploads/, so the example imported cleanly
yet still raised FileNotFoundError for a normally uploaded file. Use
deerflow.uploads.manager.get_uploads_dir(), which resolves the effective user
and validates the thread id.

* docs(backend): show the real uploads path layout in every PATH_EXAMPLES demo

The remaining sections still documented the pre-isolation
threads/<thread_id>/ bucket and called `path` a backend-relative string, while
the upload and list APIs return an absolute path under
users/<user_id>/threads/<thread_id>/user-data/uploads/. Reproduced with
deerflow.uploads.manager.get_uploads_dir() and
uploads._list_uploaded_files_for_thread() against DEER_FLOW_HOME: the listed
path is /<home>/users/default/threads/abc123/user-data/uploads/report.pdf.

* docs(backend): clarify runtime home precedence

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-25 14:15:59 +08:00
Hyeonsang Cho
3448c3426e
fix(config): honor $VAR substitution for request_admission integer fields (#5838)
* fix(config): honor $VAR substitution for request_admission integer fields

AppConfig.resolve_env_variables replaces every `$VAR` reference with the raw
environment string. RequestAdmissionConfig.requests_per_minute and
max_queue_size are declared `strict=True` so a bool or float cannot slip in,
but strict mode also refuses the string "60", so `requests_per_minute: $RPM`
failed the whole config load with "Input should be a valid integer" even when
RPM=60 — contradicting config.example.yaml's promise that environment
variables work for every field. They were the only strict fields in the
config schema; every other integer coerces in lax mode.

Wrap both fields in a BeforeValidator that converts a decimal integer literal
arriving as a string and leaves everything else untouched, so the strict check
behind it still rejects bools, floats, "60.0" and non-numeric strings.

Tests pin the accepted literals, the rejections that must survive, and the
end-to-end path from a config.yaml `$VAR` reference through AppConfig.from_file.

* docs(changelog): reference #5838 in the request_admission entries
2026-09-25 14:06:33 +08:00
yang rui
1da284bb3a
feat(prompts): add operator overlays for core system prompts (#5794)
* feat(prompts): add operator overlays for core system prompts

* test(prompts): isolate SOUL forwarding test from operator config

* fix(prompts): wrap complete subagent system message with overlay

* fix(subagents): serialize prompt overlays in durable batches

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-25 09:51:11 +08:00
spud
8f08df9b48
fix(sandbox): fence ambiguous AIO session creation ownership (#5711)
* fix(sandbox): quarantine ambiguous shell session creation

* fix(sandbox): quarantine ambiguous bash session creation

* fix(sandbox): recycle sandboxes with ambiguous session creates

* fix(sandbox): keep quarantined release teardown non-raising

* docs(sandbox): keep guidance within instruction budget

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-22 20:48:22 +08:00
spud
74f006951c
fix(sandbox): bound AIO list_dir with a directory deadline (#5662)
* fix(sandbox): bound remaining shell session cleanup requests

release_command_scope(), close()'s scoped-session drain, and close()'s recovery-session cleanup called shell.cleanup_session() without a request budget, so a stalled cleanup could hold the thread-key serializer, scoped.lock, or the sandbox lock for the SDK's default transport budget. Pass the existing _bounded_cleanup_request_options() (5s, max_retries=0) at those three sites; _cleanup_session_best_effort() and its swallow-and-log contract are unchanged, as are create_session and list_dir lifetimes.

* fix(sandbox): bound AIO list_dir with a directory deadline

Give AioSandbox.list_dir its own 60s directory deadline with a 65s no-retry
host envelope, independent of bash_command_timeout. Preserve #5634's
shell-generation selection: list_dir runs on the current recovery session once
the implicit shell is fenced, and an ambiguous list_dir outcome - transport
timeout or an ambiguous returned status - fences whichever generation actually
executed the request, dropping local recovery ownership and attempting bounded
best-effort cleanup instead of leaving that session reusable. hard_timeout
remains definite termination and keeps the targeted session reusable. Only
completed/None results are parsed, so a partial find is never returned as a
complete listing.

Session creation RPC lifetime remains out of scope.

* fix(sandbox): recover sessions after transport failures

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-22 19:49:00 +08:00
spud
61a99a1e8a
fix(sandbox): enforce bash command timeouts in AIO sandbox (#5634)
* fix(sandbox): enforce AIO shell command timeouts

Map Sandbox.execute_command(timeout=T) onto the AIO legacy shell as a server-side hard_timeout=T plus a bounded host request (ceil(T+5)s, max_retries=0), preserve and render the upstream status (hard_timeout -> terminated + Exit Code: 124; no_change_timeout -> may still be running), keep no_change_timeout from preempting hard_timeout, and restrict ErrorObservation replay to completed results.

* fix(sandbox): contain ambiguous AIO command outcomes

Treat a transport timeout and statuses terminated/no_change_timeout/unknown as ambiguous: never replay the current command, fence the scoped or recovery session generation with bounded best-effort cleanup, and never adopt a replacement session for those statuses. hard_timeout and completed keep the session reusable. Align the env-bearing bash.exec path with the same contract and make its retry result authoritative.

* fix(sandbox): wire AIO command timeout from provider config

Restore bash_tool's non-local behaviour (other providers keep their own defaults) and let AioSandboxProvider read sandbox.bash_command_timeout as the sandbox's default_command_timeout at every construction site, with explicit per-call timeouts still winning. Widen the config field to float with allow_inf_nan=False and read the provider value with the module fallback so partial-config providers keep working.

* docs(sandbox): document AIO command timeout semantics

Document the provider-scoped deadline (LocalSandbox/AioSandbox/OpenSandbox wire it; other providers keep their defaults), the semver-vs-frozen-:latest image behaviour, the no-replay guarantees, and trim the sandbox guidance back under its size budget.

* docs(sandbox): scope the wedge-resistance claim to command requests

willem-bd's review on #5634 reproduced that list_dir and both create_session call sites still make unbounded SDK requests while holding the same sandbox lock, so the guide must not read as if every AIO request is bounded. State that the bounded T+5s host wait applies to command requests and name session-creation and file/list RPCs as not covered; bounding those remains a separate operation-deadline/control-plane follow-up.

* fix(sandbox): make list_dir honor the shell generation fence

list_dir previously always targeted the implicit persistent shell, so after an ambiguous command outcome fenced that generation (#5634) a listing could re-enter it. Route default-shell target selection through _ensure_default_shell_session_id so both execute_command and list_dir reuse the explicit recovery session, creating one only when the implicit shell is fenced. No list_dir deadline, hard_timeout, request budget, retry, or status semantics are added; those remain scoped to #5644.
2026-09-22 11:29:38 +08:00
zhangwei-way
b6503e9a35
feat(knowledge): add per-message RAGFlow retrieval scope (#5238)
* feat(knowledge): integrate RAGFlow retrieval and management

* test(knowledge): cover merged listing tool

* feat(knowledge): add per-message retrieval scope

* chore(docs): remove unrelated document

* docs(knowledge): add interaction screenshots

* feat(knowledge): simplify scope selector trigger

* docs(knowledge): refresh selector screenshot

* feat(knowledge): defer standalone management

* docs(knowledge): show chat-only scope UI

* fix(knowledge): honor scope on clarification replies

* fix(knowledge): harden scoped replay validation

* docs(knowledge): clarify replay scope precedence

* fix(knowledge): keep provider settings on tools

* fix(config): preserve tools-only knowledge settings

* fix(knowledge): submit custom assistant identity

* refactor(knowledge): trim PR scope changes

* fix(knowledge): sanitize document scope display

* feat(knowledge): enable scope selection in main chat

* fix(knowledge): emphasize active scope icon without button frame

* fix(knowledge): close context scrubbing and refresh e2e checks

* fix(knowledge): preserve idempotent canonical retries

* fix(knowledge): accept promptless conversation runs

* style(knowledge): format backend regression tests

* chore(knowledge): trim PR scope and fix frontend format

* fix(knowledge): remove shared-scope notice

* fix(knowledge): remove scope persistence notice

* docs(knowledge): include main chat in catalog scope

* fix(knowledge): preserve scope recovery and upgrades

* fix(config): preserve LightRAG knowledge upgrades

---------

Co-authored-by: foreleven <for-eleven@hotmail.com>
2026-09-18 16:59:31 +08:00
Weng Qiang
a23dbdd837
feat(tavily): support configured search domain filters (#5513) 2026-09-18 07:25:49 +08:00
Dan Caldr
f2857ad3ff
fix(sandbox): default to loopback bind on Docker Desktop for DooD sandboxes (#5446)
* fix(sandbox): default to loopback bind on Docker Desktop for DooD sandboxes (#5445)

* fix(sandbox): memoize desktop detection and clarify bind host docstring (#5445)

* fix(sandbox): latch desktop detection on success only to permit retry on transient failure (#5445)

* fix(sandbox): restrict desktop loopback bind to local DooD hostnames (#5445)

* fix(sandbox): add Desktop legacy aliases and parametrize DooD host tests (#5445)
2026-09-17 07:55:22 +08:00
Onefly
d8d110c637
fix(sandbox): prevent AIO subagent session eviction (#5178)
* fix(sandbox): prevent AIO subagent session eviction

* fix(sandbox): address PR 5178 review issues

* fix(sandbox): handle transient session and metadata failures

* fix(sandbox): fence capacity upgrades and validate reused limits

* docs(sandbox): restore list indentation and trim guidance

* fix(ci): stabilize Buzz persistence test and trim sandbox guidance

---------

Co-authored-by: ranxi2001 <ranxi2001@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-17 07:45:19 +08:00
Totoro
5b591a9039
feat(gateway): accept conversation references in run context and report the capability (#5463)
* feat(gateway): accept conversation references in run context and report the capability

LangGraph SDK clients build a fixed run body and drop unknown top-level
fields, so they cannot send the conversation_references field from #5399.
RunCreateRequest now lifts context.conversation_references into the
top-level field before validation, so it keeps the same bounds and error
locations, and drops it from context, so it never reaches the merged run
context or the checkpointed configurable. Sending both is a 422.

GET /api/features reports conversation_references {enabled, max_references}
with the same "tool is configured" predicate as run admission, so a client
can hide an entry point on deployments without the tool.

Related to #5398.

* fix(gateway): report the field type error for a malformed top-level reference list

A malformed top-level conversation_references sent alongside a context
list now fails with the field's own type error instead of the conflict
message. The tool-configured predicate reads tool.use directly, and the
features test doubles carry that attribute like every real ToolConfig.

* fix(gateway): treat every list-like top-level reference value as a conflict

Pydantic's lax mode coerces tuples, sets, frozensets and deques into the
list[str] field, so a direct Python caller passing one of those together
with context.conversation_references now reports the conflict instead of
slipping both grants through. Unreachable over HTTP, where JSON has no
such types.

* fix(gateway): ask pydantic whether a top-level reference value is list-like

Enumerating list-like types cannot track pydantic's lax acceptance set
(generators, UserList, dict key views also coerce into list[str]). The
conflict guard now validates the top-level value with a TypeAdapter for
list[Any]: whatever pydantic would coerce reports the conflict when it is
non-empty, and whatever it rejects still surfaces the field's own type
error. Regression tests cover deque, UserList, dict keys and a generator,
plus rejected scalars.

* fix(gateway): probe top-level references with the field's own annotation

The conflict guard now validates the top-level value with the exact item
annotation the field uses, so its acceptance set is the field's rather than
a superset: an item the field rejects (an empty string, a non-string, the
ints of a range or dict view) surfaces the field's own item error instead
of a conflict. The annotation is shared through one alias so the two cannot
drift.

* fix(gateway): materialise a one-shot iterator before probing top-level references

The item-validating probe could consume a generator while collecting an
item error, after which the field re-validated the exhausted iterator,
coerced it to [] and let the request through with the key still in
context. Iterators are now read once into a list that both the probe and
the field validate, so a bad item is reported at its index and a valid
generator is kept.

* fix(gateway): materialise every once-walkable iterable before probing references

Pydantic coerces any iterable into the list field, and an object whose
__iter__ hands out a generator once is not an Iterator instance, so the
previous gate let it reach the probe and be consumed. The lift now reads
every iterable except lists, tuples and the shapes the field rejects as a
whole (str, bytes, dict) into a list first, so the probe and the field
always validate the same items.

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-09-16 15:37:57 +08:00
yang rui
3dc895df4d
feat(models): pace shared RPM budgets before dispatch (#5432)
* feat(models): add shared RPM admission queues

* fix(models): address admission pacing review feedback

* docs: simplify request admission quick start guidance

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-15 07:14:35 +08:00
Totoro
6177b07c06
fix(conversation): clarify reference semantics and keep reader pages inline (#5421)
Document that read permission expiry and source deletion do not erase
text already copied into the destination conversation, and that reads
follow the source's current visible history. Truncated results now tell
the agent to acknowledge the omission and ask for the missing material
before claiming every requirement is covered.

Pages were filled to 20,000 text characters by cutting the last message
that did not fit, and that suffix could never be paged back. They could
also exceed the default 12,000-character tool-output budget, which
externalized the page to a file. Pages are now sized by their serialized
length against the read_conversation tool-output budget; a message that
does not fit starts the next page intact, so only a message over 4,000
characters (or one whose escaped JSON alone exceeds the budget) is cut.

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 15:05:30 +08:00
Totoro
533e30e7f2
[feat] add opt-in conversation reads for Gateway runs (#5399)
* feat: add scoped conversation reads to gateway runs

* refactor: share conversation tool path and reuse parsed text

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-09-14 10:51:01 +08:00
Hyeonsang Cho
1dd48d14d2
fix(models): reuse the Claude Code OAuth token read from a file descriptor (#5411)
* fix(models): reuse the Claude Code OAuth token read from a file descriptor

ClaudeChatModel accepts a Claude Code OAuth token through
CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR, but that handoff can be drained
only once: a pipe returns EOF after the first read and a file descriptor
keeps its advanced offset. _read_secret_from_file_descriptor read it again
on every call and kept nothing. Every ClaudeChatModel instance loads
credentials in model_post_init, and create_chat_model builds fresh
instances per run, so with a descriptor-only handoff the first model
authenticated and every model after it -- including the title model of
the very first run -- had no credential. The Anthropic SDK then raised
"Could not resolve authentication method" before sending a request.

A secret read from a descriptor is now kept for the life of the process,
keyed by (env_var, fd), so a different descriptor is still read fresh.
The read happens under a lock so two threads building their first model
concurrently cannot race one of them to EOF. Empty reads and OSError are
not cached and behave as before; lookup order, config keys, and log
messages are unchanged.

* docs(changelog): reference #5411 in the Claude Code OAuth descriptor fix entry

* test(models): pin that a closed descriptor handoff keeps its token

Review follow-up on #5411: the descriptor secret cache is keyed on the
fd number, which the OS recycles. Folding os.fstat identity into the key
would break the property the cache exists for -- once the handoff fd is
closed after the first read, fstat raises EBADF and every later model
would lose the token again -- and it would still miss a regular file
rewritten in place, which keeps its st_dev/st_ino.

The handoff is fixed at process start, so keep the number as the key and
state the invariant instead: a closed handoff keeps serving its token, a
secret placed on a recycled number is not re-read, and anything handing
over a new secret in-process must clear the cache. A new test pins the
closed-handoff behavior; an fstat-fingerprinted key fails exactly that
test.
2026-09-14 09:23:12 +08:00
Xuehao Xu
56540fab01
fix(gateway): make recursion limit configurable (#5390)
* fix(gateway): make recursion limit configurable

* docs: keep backend guidance within inherited size budget

* fix(gateway): address recursion limit review feedback

* docs(gateway): clarify recursion default scope

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-13 17:08:33 +08:00
wutongyuonce
48a8978b7b
feat(scheduler): add interval schedule type (#5291)
* feat(scheduler): add interval schedule type

Allow scheduled tasks to fire every N seconds from last dispatch, not
only wall-clock cron or a single run_at. Cadence is UTC now+N with no
missed-beat catch-up, bounded by min_once_delay_seconds and 30 days.

* fix(scheduler): let interval tasks create, edit, and keep next run

Create/edit now keep every_seconds. Unchanged interval spec no longer
resets next_run_at, including timezone-only PATCH.

* fix(scheduler): keep non-minute intervals on edit

Stop rounding every_seconds to whole minutes in the form. Values that
are not whole minutes or hours now use a seconds unit so edit/duplicate
round-trips the stored cadence instead of rewriting it and resetting
next_run_at. Document that min_once_delay_seconds is also the interval
floor.

* fix(scheduler): clamp interval seconds to the default 60s floor

The new seconds unit allowed 1–59, which the API rejects under the
default min_once_delay_seconds. Clamp the form to >= 60 and show the
floor next to the preview. Also mention interval in the scheduler
field_doc, matching config.example.yaml.

* fix(scheduler): do not clamp interval amount while typing

Keystroke clamp made 90 become 9 -> 60, then 600, and backspace could
not leave 60. Keep the raw field text and apply the 60s floor on blur
and emit only.

* test(scheduler): cover interval input editing

* fix(frontend): preserve saved interval cadence until edited

* style(tests): format scheduled task router tests

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-10 16:46:01 +08:00
wutongyuonce
69f0f483eb
feat(scheduler): let scheduled tasks pin a custom agent (#5288)
* feat(scheduler): let scheduled tasks pin a custom agent

Create and update accept optional assistant_id, defaulting to lead_agent.
Custom names are normalized and must already exist for the task owner.
The workspace form exposes the same choice, and duplicate copies it.

Fixes #5286

* fix(scheduler): keep assistant-id PR free of interval tests

Drop the six interval tests that belonged to the interval schedule PR
and fail here because this tree still only accepts once/cron.

Treat lead_agent case-insensitively so LEAD_AGENT / lead-agent store
as the default. Omit unchanged assistant_id on edit so a deleted custom
agent does not 422 unrelated PATCH (rename, reschedule).

* fix(scheduler): format task page and browser tests

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-10 15:36:49 +08:00
Yusuf Gürdoğan
23bd76046a
feat(community): add Sofya web search provider (#5239)
* feat(community): add Sofya web search provider

Add a community provider backed by Sofya (https://sofya.co). Its search
endpoint returns the content of the result pages, not only their snippets,
and its fetch endpoint returns a page as markdown. Both are plain JSON over
HTTP, so this needs no extra Python package (uses httpx, already a
dependency).

Changes:
- backend/packages/harness/deerflow/community/sofya/__init__.py
- backend/packages/harness/deerflow/community/sofya/tools.py
  Implements web_search_tool and web_fetch_tool using httpx.
  API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY
  env var. Follows the same interface and output shape as the existing
  ddg_search and serper providers, including the max_results parameter with
  config override and the structured "No results found" error.
- backend/tests/test_sofya_tools.py
  Unit tests covering API key resolution, config overrides, result mapping,
  time range, HTTP errors, empty results, and fetch failures.
- config.example.yaml: add commented-out Sofya web_search and web_fetch
  examples alongside the other providers
- .env.example: add SOFYA_API_KEY placeholder
- backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and
  the environment variables

* fix(sofya): honor caller max_results, validate search_depth, join time_range contract test

- Caller-supplied max_results now wins; config is used only when the
  argument is omitted, matching GroundRoute.
- search_depth is clamped to basic/snippets; an unsupported value logs a
  warning and falls back to basic.
- Sofya added to the shared time_range schema contract test.

* fix(sofya): cap per-result content so a search stays inline

An unbounded search payload (up to 20 read pages) crossed the tool output
budget middleware's externalize_min_chars threshold, which replaces the
result list with a file reference. Cap each result's content at
contents_max_characters (default 2000, 0 disables), matching Exa's config
key. Five capped results stay under the 12000 char threshold.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* fix(sofya): list Sofya in the recency contract, coerce non-string content

_clip subscripted its input, so a non-string content or description from
the API raised TypeError instead of degrading. Coerce to text first, the
way _sofya_post and _response_results guard the shapes around it. Also add
Sofya to the Web Search Recency section in backend/AGENTS.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog

web_fetch sliced its content the same way web_search did before the last
push: a truthy non-string from the API passed the falsiness guard and then
raised TypeError. Reuse _clip, keeping the `or ""` so empty content still
reports "No content found".

Also add sofya to the community provider inventory in
packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs

The ZH web_fetch tab list named five providers where EN names seven. Both
tabs mirror their EN counterparts, so the two locales list the same
web_fetch providers again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-08 10:50:03 +08:00
hataa
aec7d73890
feat(knowledge): add read-only LightRAG retrieval, fixes #5208 (#5209) 2026-09-06 10:21:58 +08:00
Aari
0f7d8709d3
feat(sandbox): add controlled egress with approvals (#5152)
* feat(sandbox): add controlled egress approvals

* Apply batched suggestions from code review

* fix(sandbox): harden restricted network policy

* fix(sandbox): harden denied egress handling

* fix(sandbox): isolate network proxy sidecar

* chore: retry sandbox image smoke

* fix(sandbox): close remaining network policy gaps

* fix(sandbox): harden relay token rejection

* fix(sandbox): fence incompatible policy replacement

* fix(sandbox): replace containers across network modes

* fix(sandbox): close remaining lifecycle gaps

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-04 23:46:57 +08:00
Jun
83cb6767b3
fix(sandbox): add FOWNER for AIO 1.11 startup (#5163)
* fix(sandbox): add FOWNER for AIO 1.11 startup

* test(sandbox): cover FOWNER startup capability

* docs(sandbox): document FOWNER capability

* test(sandbox): pin FOWNER regression smoke

* ci(sandbox): allow pinning FOWNER smoke image

* style(sandbox): format FOWNER smoke test

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-04 00:03:49 +08:00
Aari
317577e285
fix: enforce custom agent skill allowlists in sandboxes (#5077)
* fix: enforce agent skill allowlists in sandboxes

* fix: guard E2B skill projection resets

* fix: preserve agent skill isolation across delegation

* fix: close sandbox skill isolation bypasses

* fix(sandbox): close skill isolation review gaps

* fix(sandbox): harden skill isolation lifecycle
2026-08-31 14:35:08 +08:00
luo jiyin
0dd233afc4
feat(sandbox): make E2B mount upload deadline configurable (#4876)
* feat(e2b-sandbox): make mount upload deadline configurable

Replace the hardcoded 120-second mount upload deadline with a
configurable `mount_upload_deadline_seconds` key read from
SandboxConfig (extra=allow). The value is validated: zero and
negative inputs are clamped to 1 second. Omitting the key
preserves the existing 120-second default.

This addresses the follow-up from PR #4842 review: operators
with large mounts or slow networks can now size the deadline to
their deployment without changing code.

* fix(e2b-sandbox): address review feedback on configurable deadline

- Remove import-time default capture from _mount_deadline_reason()
  and _MountUploadBudget.deadline_seconds to prevent silent drift.
- Add warning log when mount_upload_deadline_seconds is clamped to 1
  (was silent before).
- Update AGENTS.md E2B Mount Uploads section: deadline is now
  configurable, not fixed 120.
- Add mount_upload_deadline_seconds to YAML examples in provider
  docstring and __init__.py.
- Add config-path test that exercises SandboxConfig -> _load_config ->
  _apply_mounts end-to-end.

* fix(e2b-sandbox-provider): handle non-numeric mount_upload_deadline_seconds

Guard _resolve_mount_upload_deadline against None, non-numeric strings,
and other invalid values. None returns the default; non-numeric strings
like '120s' or 'abc' log a warning and fall back to the 120-second
default instead of crashing provider init with TypeError/ValueError.

Extend the parametrized clamp test with None, suffix, and alpha cases,
and add a warning assertion. Update CONFIGURATION.md with the new
mount_upload_deadline_seconds key and its behavior.

* fix(sandbox): handle infinite mount deadline
2026-08-30 11:31:53 +08:00
Serply
4dbfe37ff3
feat(community): add Serply web search tool (#5023)
Add deerflow.community.serply.tools:web_search_tool, a Google SERP
provider for the web_search slot that also covers Google News and Google
Scholar through an optional `vertical` config option. Reads the key from
api_key in config.yaml or SERPLY_API_KEY, clamps max_results to Serply's
1-100 range, and returns the same structured JSON errors as the Serper
and Brave tools.

Register the provider in config.example.yaml, scripts/doctor.py,
scripts/wizard/providers.py, .env.example, backend/docs/CONFIGURATION.md,
the en/zh tools.mdx provider tabs, and tools/AGENTS.md. Tests mock httpx.
2026-08-28 10:30:30 +08:00
陈志谦
9e2c1be697
fix(sandbox): harden local Docker sandbox containers and port binding (#4986)
* fix(sandbox): harden local Docker sandbox containers and port binding

Root causes (security audit SBX-1/SBX-2) in the local container backend:
- _resolve_docker_bind_host published sandbox ports on 0.0.0.0 whenever
  DEER_FLOW_SANDBOX_HOST was non-loopback (docker-compose defaults to
  host.docker.internal), exposing the unauthenticated /v1/shell/* exec
  API on every host interface.
- _start_container ran every sandbox with seccomp=unconfined and no
  capability, privilege-escalation, or resource limits, so untrusted
  model-authored code could exhaust the host, escalate privileges, and
  reach internal networks / cloud metadata endpoints directly.

Hardening changes and defaults:
- Port binding: non-loopback sandbox hosts now bind the Docker default
  bridge gateway instead of 0.0.0.0, discovered dynamically via
  `docker network inspect bridge` with a static 172.17.0.1 fallback.
  host.docker.internal resolves to that gateway through host-gateway,
  so DooD gateways and the Docker host still reach the sandbox while
  external interfaces no longer see the port.
  DEER_FLOW_SANDBOX_BIND_HOST=0.0.0.0 restores the legacy broad bind.
- seccomp=unconfined is no longer unconditional: sandboxes run with
  Docker's default seccomp profile; opt back in with
  DEER_FLOW_SANDBOX_SECCOMP_UNCONFINED=1, only when the sandbox image
  is verified to require syscalls the default profile blocks.
- Add --cap-drop=ALL and --security-opt no-new-privileges (Docker only;
  the Apple Container CLI does not support these flags).
- Bounded resources with env overrides: --memory 2g
  (DEER_FLOW_SANDBOX_MEMORY), --cpus 2 (DEER_FLOW_SANDBOX_CPUS),
  --pids-limit 512 (DEER_FLOW_SANDBOX_PIDS_LIMIT); each also accepts
  "0"/"none" to disable the limit.
- No --user is forced by default (the default AIO sandbox image's user
  is upstream-controlled and unverified), but
  DEER_FLOW_SANDBOX_CONTAINER_USER passes one through for deployments
  that know their image.
- DEER_FLOW_SANDBOX_NETWORK passes --network so sandboxes can be
  attached to a dedicated egress-controlled network; default networking
  is unchanged.

backend/docs/CONFIGURATION.md documents the new bind behavior and every
override; tests cover each default and escape hatch.

* fix(sandbox): follow host-gateway mapping for binds; keep image-required seccomp default

Review follow-ups on the hardening change:

- Bind: resolve the sandbox host itself and bind that address, instead of
  assuming the default bridge IPv4. host.docker.internal follows the
  daemon host-gateway-ip mapping (customizable, possibly IPv6), so the
  resolved address is exactly where the gateway connects — the published
  port and advertised URL always match. IPv6 is bracketed for docker -p,
  zone ids stripped, wildcard resolutions ignored; unresolved hosts fall
  back to the bridge gateway with a warning pointing at
  DEER_FLOW_SANDBOX_BIND_HOST.
- seccomp: the shipped AIO image needs seccomp=unconfined for its
  Chromium browser (upstream quick-start always passes it; the upstream
  FAQ documents the browser failing under Docker default profile), so
  that option returns as the default. Tightening stays possible via
  DEER_FLOW_SANDBOX_SECCOMP_PROFILE=<path to a restricted,
  Chromium-compatible profile> or DEER_FLOW_SANDBOX_SECCOMP_UNCONFINED=0
  for images verified to work with Docker's default profile.
- cap-drop/no-new-privileges and the resource limits are unchanged.
- Tests updated for both behaviors; 37 pass.

* fix(sandbox): bracket bare IPv6 bind overrides; state seccomp default accurately

DEER_FLOW_SANDBOX_BIND_HOST was returned verbatim, so a bare IPv6 literal
like fd00::1 produced an invalid publish spec (fd00::1:port:8080); Docker
requires the bracketed form. Normalize raw and already-bracketed IPv6
literals (IPv4/hostnames untouched), with resolver-level and argv-level
tests covering the explicit IPv6 override.

The CONFIGURATION.md overview claimed Docker's default seccomp profile
stays active, contradicting the seccomp=unconfined default the table (and
the code) actually ship for the Chromium-based image; spell out the relaxed
default and where to change it.

* style(sandbox): apply ruff format to local_backend

* fix(sandbox): reject host networking, force builtin seccomp opt-out, resolve hostname binds

Review follow-up on #4986 (willem-bd):

- P1: DEER_FLOW_SANDBOX_NETWORK=host (and container:<name>) now raise a
  RuntimeError at start instead of silently voiding the hardened port
  bind — Docker discards -p/--publish in host mode and shares the
  network namespace for container:<name>, which would re-expose the
  unauthenticated exec API on the host's interfaces. Two regression
  tests cover both rejections.
- P2: the seccomp opt-out now passes seccomp=builtin explicitly instead
  of omitting the option, so a daemon configured with an unconfined or
  custom default cannot weaken the documented opt-out; the test asserts
  the flag.
- P2: hostname values in DEER_FLOW_SANDBOX_BIND_HOST resolve to an
  address before use (Docker publish specs require an IP literal as the
  host part, so host.docker.internal previously produced an invalid
  spec that prevented every sandbox from starting); unresolvable names
  raise a clear configuration error. Tests cover resolution and
  rejection; CONFIGURATION.md updated for all three behaviors.

43/43 pass in tests/test_aio_sandbox_local_backend.py; ruff check +
format clean.

* fix(sandbox): reject DEER_FLOW_SANDBOX_NETWORK=none (loopback-only, breaks published API port)

* fix(sandbox): validate the effective Docker network target; normalize IPv6 sandbox hosts once

name=host / name=none dodge raw-string checks but attach like the bare
words; strip name= prefixes and validate the effective target (network IDs
keep passing). Bracketed IPv6 sandbox hosts now resolve for the bind and
bare IPv6 hosts produce bracketed URL authorities — both input forms give
identical bind and URL addresses.

* fix(sandbox): parse the full Docker network long syntax before validating

Docker accepts comma-separated key=value fields in any order (name=, gw-priority=,
alias=, ...); a name=host field hides the host network behind surrounding fields.
Parse the CSV and validate the parsed name= target (last occurrence wins, fields
lowercased, mirroring opts/network.go); no-name values fall through like Docker's
own rejection.

* fix(sandbox): keep CHOWN/SETUID/SETGID through cap-drop=ALL for the default image

The shipped image's entrypoint starts as root, creates the gem user,
chowns /opt/jupyter and drops to that user via su; without those three
capabilities the set -e script dies before the readiness endpoint exists.
no-new-privileges stays (it blocks gaining privileges via exec, not using
the added caps). Adds a docker-gated real-image startup smoke test.

* fix(sandbox): let pre-initialized non-root images drop the startup capabilities

The CHOWN/SETUID/SETGID re-add only exists for the shipped image's root
entrypoint handoff. A custom image that never runs as root gets an explicit
opt-out (DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS=0) so those capabilities are
not left available to sandboxed code (chown on bind mounts, UID/GID
impersonation).

* test(sandbox): gate the real-image smoke test behind the live marker

The default offline suite (make test = -m 'not live') must not depend on a
third-party registry: mark the smoke test live, probe the daemon inside the
test body (never at collection time), and allow pinning the image reference
via DEER_FLOW_SANDBOX_SMOKE_IMAGE for a dedicated integration job.

* test/docs: isolate DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS in tests; add table row; split custom-image guidance

_clear_hardening_env now clears the new knob so a developer shell or .env
preset cannot flip the default-path tests. CONFIGURATION.md gains the table
row, and the custom-image guidance becomes its own paragraph with the
no-new-privileges scope stated correctly (it does not mitigate the retained
CAP_SETUID/SETGID risk).

* test(sandbox): make the live smoke test diagnosable

300s readiness budget (cold pull + cold start must not be conflated with
broken capabilities) and dump the container's last 40 log lines on failure
so the next live run tells us whether the capability set is incomplete
(chown/useradd/su errors) or the services are merely slow.

* test(ci): align the smoke test with the 60s provider deadline; add a dedicated live smoke workflow

Single-source the readiness deadline as SANDBOX_LOCAL_PROVIDER_READY_TIMEOUT
(used by both provider paths and the smoke test) so the validation cannot
drift from the production contract again. New sandbox-image-smoke.yml runs
the live test on a dedicated job, with the image reference pinnable via the
SANDBOX_SMOKE_IMAGE repository variable (digest resolved and recorded in the
job summary when falling back to :latest).

* test(sandbox): pull the failing program's own logs on smoke failure

supervisord only surfaces exit codes in docker logs; nginx's stderr lands in
files inside the container. Dump supervisor program logs, nginx -t, and the
nginx error log on failure so the next run names the exact broken line.

* ci(sandbox): export an immutable repo@digest reference for the smoke run

docker pull once on the runner platform, resolve RepoDigests[0], and pass
that immutable reference to the test via GITHUB_ENV — the recorded and
executed images can no longer diverge when the tag moves, and platform
selection is left to the daemon instead of jq over the manifest index.

* fix(sandbox): add DAC_OVERRIDE — the root nginx master writes gem-owned logs

The image's root nginx master opens /var/log/nginx/{access,error}.log,
which belong to the gem user, for the container's lifetime; without
CAP_DAC_OVERRIDE it dies with 'open() failed (13: Permission denied)' on
every start (FATAL under supervisord) and readiness never arrives. Four
capabilities now: CHOWN/SETUID/SETGID for the entrypoint handoff plus this
runtime log-write need.
2026-08-27 22:53:47 +08:00
yong
846c716523
feat(search): add Tencent Cloud WSA provider (#5057)
* feat(search): add Tencent Cloud WSA provider

* docs: restore README to upstream

* docs: remove README changes from WSA provider PR

* fix(doctor): validate Tencent WSA API key
2026-08-27 18:12:05 +08:00
zhangwei139623
431892e16a
feat(knowledge): add read-only RAGFlow retrieval (#4955)
* feat(knowledge): add read-only RAGFlow retrieval

* test(knowledge): cover RAGFlow retrieval contracts

* docs(knowledge): document retrieval-only RAGFlow setup

* refactor(knowledge): move RAGFlow settings to tool config

* fix(ragflow): bind retrieval to configured datasets

* docs(ragflow): record validated response versions

* fix(ragflow): bind retrieval by dataset id

* fix(ragflow): search all datasets by default

* fix(ragflow): retrieve mixed embeddings by group

* docs(ragflow): keep feature details out of agent guides

* docs(ragflow): remove agent guide changes

* docs(ragflow): remove root readme changes

* fix(ragflow): handle unresolved and empty datasets

* fix(ragflow): harden dataset scope and errors
2026-08-25 16:57:55 +08:00
Aari
645ca08f16
fix(scheduler): enqueue busy scheduled task runs (#4918)
* fix(frontend): clarify reuse-thread scheduling behavior

* fix(scheduler): enqueue overlapping scheduled runs

* fix(scheduler): preserve queue lease fencing

* fix(scheduler): close queue concurrency races

* fix(scheduler): harden queue timeout bookkeeping

* fix(scheduler): preserve manual failure schedule

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-08-24 15:38:32 +08:00
Aleksandr Sapronov
613b90b0e6
feat(scheduler): make scheduled-run recursion_limit configurable (#4848)
Adds scheduler.recursion_limit to config.yaml (default 1000, clamped by
max_recursion_limit) so scheduled background runs can use a different
recursion limit than the web UI. The value is read at dispatch time, so
a YAML edit applies to the next scheduled run without a Gateway restart.

Also logs a warning when the resolver falls back to the default or
clamps the configured value.
2026-08-24 07:09:08 +08:00
muguo
917fe595fc
feat(sandbox): add OpenSandbox provider (#4877) 2026-08-23 15:46:10 +08:00
DanielWalnut
062ba9ddfc
feat: integrate MiniMax Code as a native ACP agent (#4846)
* feat: integrate MiniMax Code as an ACP agent

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-08-17 08:16:14 +08:00
rain02333z-spec
828363705a
fix(scheduler): support safe multi-instance scheduler recovery (#4713)
* fix(scheduler): reject unsafe multi-worker startup

* fix(scheduler): support safe multi-instance recovery

* fix(scheduler): make multi-instance recovery lease-safe

* fix(scheduler): address multi-instance review feedback

* docs(scheduler): add multi-instance upgrade notes

---------

Co-authored-by: rain02333z-spec <225106191+rain02333z-spec@users.noreply.github.com>
2026-08-14 23:50:40 +08:00
阿泽
9c7cd4cad3
feat(sandbox): add thread data mount override for upload sync (#4536) 2026-07-28 23:41:14 +08:00
Aari
a5059b8284
fix(subagents): isolate callbacks and activate skills lazily (#4497) 2026-07-28 23:29:14 +08:00
March-77
b22f85c686
fix(sandbox): reconcile E2B sandboxes safely (#4443)
* fix(sandbox): reconcile E2B sandboxes safely

* fix(sandbox): clear failed E2B adoption intent
2026-07-27 14:10:24 +08:00
March-77
2e5c8da257
fix(sandbox): bypass proxies for local AIO traffic (#4444)
* fix(sandbox): bypass proxies for local AIO traffic

* fix(sandbox): classify public IPv6 proxy targets
2026-07-27 07:47:39 +08:00
Vanzeren
1c7531242c
feat(runtime): record terminal artifact delivery receipts (slice 1 of #4272) (#4365)
* feat(runtime): record terminal artifact delivery receipts (#4272)

* fix(runtime): persist delivery receipts across recovery

* test(runtime): cover delivery receipt invariants

* fix(runtime): preserve terminal status on receipt outages
2026-07-26 21:45:47 +08:00