* fix(community): refuse non-global addresses in the shared SSRF guard
is_blocked_address ORed is_private, is_loopback, is_link_local,
is_reserved, is_multicast and is_unspecified. None of them covers the
100.64.0.0/10 shared address space, which holds CGNAT and Tailscale
hosts and Alibaba Cloud's 100.100.100.200 instance metadata endpoint, so
web_fetch, web_capture, the agentic browser and personal MCP connections
could reach it, including through the IPv4-mapped ::ffff:100.100.100.200
form a DNS answer can carry.
Also refuse any address that is not is_global, the classification the
repo's other SSRF filters already use. Keep the existing flags alongside
it: some non-public forms still report is_global, such as the NAT64
spelling of a metadata address (64:ff9b::a9fe:a9fe) and 224.0.0.1, so an
is_global-only check would newly allow them.
* docs(changelog): reference #6202 in the shared address space entry
* test(community): pin the IPv4-mapped metadata spelling at the web_fetch and personal MCP entry points
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tui): preserve multiline pasted input
The TUI composer used a single-line `Input`, so a bracketed paste
containing code, a stack trace, or multiple paragraphs kept only the
first line when submitted. The composer is now a multiline `TextArea`:
Enter still submits, Up/Down move inside the document before falling
back to input history at the first/last line, and the slash palette is
unchanged.
Originally #4630 by @felix-windsor, cherry-picked onto current main;
conflicts in app.py key gating (transcript paging keys), test imports,
and the relocated TUI docs were resolved by hand.
Fixes#4613
* fix(tui): submit multiline input unchanged and document the widgets layer
Address the review on #4630:
- Send non-command input to the agent and history exactly as typed or
pasted. Only the empty-input check uses strip(), so leading
indentation and trailing newlines in pasted code and stack traces
survive. Slash commands are still trimmed so they resolve as before.
- widgets/composer.py imports Textual, so the module-layout notes in
tui/AGENTS.md and docs/TUI.md now exempt widgets/ alongside app.py.
- Cover pressing Down on the last document line (history fallback),
whitespace-only input, and slash commands with surrounding whitespace.
Co-authored-by: Felix <felixwindsor3344@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(tui): treat soft-wrapped rows as composer navigation boundaries
TextArea soft-wraps long lines, and its Up/Down move between the wrapped
rows of one logical line. The history fallback only looked at logical
rows, so Up from the second visual row of a long prompt recalled history
instead of moving the cursor, and Down from its first visual row could
replace the draft with history's empty value.
Use the TextArea navigator's first/last wrapped-row checks for the
boundaries, and cover wrapped prompts in both directions. Also restore a
tui/AGENTS.md sentence about PageUp/PageDown to the app.py entry it
belongs to.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Felix <felixwindsor3344@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(community): add bounded retries for Jina web fetch
Closes#6116
Keep retries opt-in and share a total deadline across requests and backoff. Preserve cancellation and the existing Error contract. Include offline DNS and local-wheel fixture corrections needed to run the complete backend suite in the isolated verifier.
Signed-off-by: 97three <2212371308@qq.com>
* docs(community): scope Jina retry guidance to its provider
Signed-off-by: 97three <2212371308@qq.com>
* fix(community): jitter bounded Jina retry backoff
* docs(community): apply provider-scoped Jina guidance
Signed-off-by: 97three <2212371308@qq.com>
---------
Signed-off-by: 97three <2212371308@qq.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Checkpoint durable refs for host-externalized tool results and materialize verified bytes before model calls on another Gateway.
Refs: #4189
Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
* feat(persistence): add notification delivery outbox for scheduled task IM push
New notification_deliveries table and repository (migration 0012) backing
the outbox pattern for issue #4254: idempotent enqueue keyed on task_run_id
plus event plus provider plus target, atomic claim of due rows, and
exponential-backoff retry state moving pending to sending to sent or failed.
Execution status stays in scheduled_task_runs; delivery status lives here.
* feat(scheduler): enqueue and deliver scheduled-task notifications over IM channels
Closes the loop for issue #4254. On run completion the scheduler writes
outbox rows for the task owner's connected channel identities (success and
failure events; interrupts stay silent), and a new NotificationDeliveryWorker
polls due rows and pushes them through the channel's proactive send path.
Channel gains send_notification; WeCom implements it via the WS client's
send_message with errcode checking, the base class raises for providers
without proactive push.
Delivery is best-effort and never shadows execution status: enqueue and send
failures are logged and recorded in the outbox, which retries with backoff.
Channel liveness is re-checked at delivery time, and the worker stops before
the channel service on shutdown. Feature activates only when channel
connections are enabled; no new dependencies.
Verified end to end against a live WeCom bot: scheduled run completion
produced an outbox row that was delivered in under one second.
* feat(scheduler): include scheduled run result summary in IM notifications
The v1 notification only carried the run outcome skeleton (status, task id,
run id). Completed runs now include the agent's final answer so the IM push
is actually useful headlessly.
- The delivery worker accepts an optional resolve_run_summary(run_id,
user_id) resolver; for run_completed events it enriches the rendered text
with a "Result:" line (bounded to 1000 chars). The lookup is best-effort:
resolver failures or missing summaries degrade to the skeleton text and
never block or fail the delivery.
- The summary is read at delivery time from runs.last_ai_message (the
convenience column written on run completion), not snapshotted into the
outbox payload; retried deliveries re-read the freshest value and the
outbox row payload stays skeleton-only. The claimed outbox row is never
mutated in place.
- Gateway wiring scopes the read to the outbox row's owner_user_id so a
stale row cannot pull another user's run content.
- Failed runs keep the error-only skeleton: a partial answer on a failed
run would be misleading.
Tests: 9 new worker cases (rendering incl. truncation, sync/async
resolvers, failure/missing fallbacks, no resolver call for failed runs).
Full scheduled + outbox + channels regression green (441 passed), ruff
check + format clean.
* fix(scheduler): address review feedback on notification delivery reliability
Review follow-ups for the scheduled-task IM notification outbox:
1. Crash recovery for "sending" rows: claim now stamps updated_at and the
worker reconciles rows stuck in "sending" past a 10-minute lease back to
"pending" before every claim. Per-row try/except isolates a crashing
delivery from the rest of the claimed batch.
2. Double-send race under multiple workers: claim returns exactly the rows
its own UPDATE...RETURNING flipped (guarded by status='pending'), so a
rival worker's concurrent claim can no longer be handed out twice.
3. Restore non-ASCII comments and drop the BOM accidentally introduced in
test_scheduled_task_service.py.
4. Manual "run now" completions no longer enqueue IM notifications; the
user is already watching the UI. Missing trigger metadata fails safe
towards notifying.
5. Channel-outage failures no longer consume the retry budget: they are
parked with count_attempt=False on a flat 15-minute backoff so an
hours-long outage cannot exhaust retries and drop the notification.
Tests: repository 16, worker 21, scheduled-task service 31; targeted
regression across notification/scheduler/channel suites: 511 passed.
* fix(scheduler): address round-2 review on IM notification delivery
Park WeCom transport/outage failures without burning retries, bound worker stop, include task titles, align enqueue with delivery, and document the IM push behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(scheduler): complete IM notification docs and drop unused list API
Add backend/channels/migrations AGENTS notes, CHANGELOG and config.example hints for the outbox push path, and remove the unused list_by_task_run repository method flagged in review.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(scheduler): address round-3 review on IM notification delivery
Cap channel-down parking by row age and parked attempts, redact IM egress
content, narrow WeCom transport exceptions, add lifespan wiring tests, and
run ruff format so lint-backend passes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(scheduler): redact entire PEM blocks in IM egress text
The previous BEGIN-only pattern left key material and the END marker in notifications.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(persistence): align 0014 parked_attempts with bootstrap schema tests
Backfill existing rows with a temporary server default, then drop it so create_all matches Alembic, and move HEAD anchors to 0014.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(persistence): format notification tests and shorten alembic revision id
ruff format --check failed on test_notification_delivery_worker.py.
0015_notification_delivery_parked_attempts exceeds alembic_version.version_num
(VARCHAR 32) and truncates on Postgres CI; use 0015_parked_attempts instead.
* test: fix post-merge CI failures after migration renumber
Update migration 0015 test HEAD expectation to 0017_parked_attempts and add queue_timeout_seconds to the lifespan notification worker mock scheduler config.
* fix(persistence): chain the notification outbox migrations after 0026
Main's head moved from 0016 to 0026_mcp_task_lease_tokens, so the two
outbox revisions are renumbered and re-parented:
- 0017_notification_deliveries -> 0027_notification_deliveries
- 0018_parked_attempts -> 0028_parked_attempts
Only the revision ids, parents and the migration guide change.
* fix(scheduler): adapt the notification hook and its docs to main
- tests: the completion tests assert on the atomic complete_run record;
new cases cover a completion that was not recorded, a task deleted
meanwhile, a failed task read, and the outbox being off.
- AGENTS.md: drop the root note and shorten the backend note to a pointer;
the two notes pushed two effective guidance chains past the 96 KiB limit.
- CHANGELOG: move the entry from the released 2.1.0 section to Unreleased
and add its link references.
* test(persistence): move the chain-head pin to the outbox migrations
test_migration_0026 pinned 0026 as the head, so it failed once the outbox
revisions chained after it. It now checks chain membership like the older
migration tests, and a new test file owns the head pin, checks the parents
and the 32-character revision id limit, and runs the upgrade and downgrade.
* fix(gateway): wire scheduled-task notifications after the channel service starts
The outbox wiring sat in the scheduler block and asked for the running
channel service there. #5035 moved the channel service start to after the
scheduler, and the two changes merged without a conflict, so on every
startup the lookup returned None and the feature disabled itself with the
"write-only outbox" warning. The lifespan tests patched the lookup to a
constant and kept passing.
The wiring now runs in its own step after the channel service start. The
scheduler is built as on main, and ScheduledTaskService gains
attach_notification_outbox() so the enqueue side is enabled once the
delivery worker is running. The guard, the worker and the shutdown order
are unchanged.
Tests: the lifespan helper can model the real accessor (None until the
channel service has started); a new case fails without this fix.
* fix(scheduler): enqueue from scheduler start and keep the push when the title read fails
Follow-up to the startup-order fix, from review.
- The enqueue side is wired with the scheduler again (it only needs the
durable table), and only the delivery worker waits for the channel
service. Attaching it later left a window after the scheduler started in
which a finished run wrote no outbox row. If the channel service is
missing or the worker cannot start, the enqueue side is detached, so the
outbox still never becomes write-only.
- A failed task read after complete_run no longer drops the notification:
the title is optional and the message falls back to the task id. A task
deleted meanwhile still stays silent.
* fix(channels): park WeCom notifications while the SDK socket is closed
The WeCom SDK raises a plain RuntimeError when its socket is not open, and
send_notification only mapped ConnectionError, TimeoutError and OSError to
ChannelUnavailable. A disconnect therefore spent the counted retry budget
and settled the row as failed after about 15 minutes, although the docs
say transport outages park without consuming retries.
send_notification now asks the client's is_connected first. The new test
uses a real, never-connected SDK client: it raised RuntimeError after three
retries before, and parks now.
* docs(scheduler): describe which outcomes notify; mirror the changelog entry
- README (en/zh) and CONFIGURATION.md: only runs that finish as success or
failed notify; occurrences that end without a finished run (launch error,
queue timeout, restart recovery) do not. Parking lasts up to about a day.
- CHANGELOG: cite the issue as text like the rest of the file, and add the
entry and link reference to CHANGELOG_zh.md.
- AGENTS.md: drop the backend pointer so this branch adds nothing to the
guidance chains closest to their size limit; the scheduler-side rules
and the lifespan order now live in app/channels/AGENTS.md.
* docs(changelog): cite #6135 next to #4843 for the notification outbox entry
* fix(scheduler,channels): re-check the IM binding at delivery time; park WeCom transport errors, count its rejections
Review round 1 on #6135.
The delivery worker now takes resolve_connections (the Gateway wires
ChannelConnectionRepository.list_connections) and, before the channel
liveness check and the summary read, requires the row's owner to still
hold a connected (provider, external_account_id). A disconnected target
is finalized with mark_failed(terminal=True); a lookup failure or an
unexpected result shape parks the row. Terminal outbox rows ignore
later status writes.
WeCom: the SDK raises RuntimeError from send_message for a socket that
is not open and for a dropped connection, but also for a platform
rejection ("Reply ack error: errcode=..."). Only that prefix keeps
counting; other RuntimeErrors, ConnectionError/TimeoutError/OSError and
websockets' ConnectionClosed park the row as ChannelUnavailable.
* fix(channels): treat a WeCom ACK rejection as proof the transport is up; pin the rejection prefix through the retry branch and the real connection-row shape
---------
Co-authored-by: GaoTianyuanhr <75788039+GaoTianyuanhr@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: GaoTianyuanhr <GaoTianyuanhr@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(uploads): reject filenames reserved for temporary staging
Guard thread and shelf upload names, validate SDK batches before copying, and preserve legacy shelf downloads. Add regression coverage and document the reserved staging namespace.
* fix(uploads): report reserved names before HTTP batch ingestion
Preflight reserved staging basenames before thread storage or sandbox setup. Return HTTP 400 with a rename hint; add HTTP and strict blocking-I/O regressions and update upload documentation.
* fix(uploads): reject Windows-style staging basenames on POSIX
Normalize backslashes before the HTTP reserved-name preflight. Add POSIX basename regressions for single and mixed batches and strict blocking-I/O coverage. Clarify the preflight scope and upload documentation.
Follow-up to the review of PR #6122; AI-assisted with OpenAI Codex.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(cli): exit non-zero when a headless run ends in an LLM error fallback
LLMErrorHandlingMiddleware turns provider failures (e.g. an expired
credential returning 401) into an AIMessage flagged
`deerflow_error_fallback` instead of raising. The Gateway run status and
the subagent executor already treat that flag as a failure. #5963 gave
`deerflow --print` / `--json` an error boundary for raised exceptions,
but a flagged fallback never raises, so both modes still printed the
fallback text and exited 0.
Both headless modes now follow #5963's failure contract when the run's
final AI message carries the flag: `--print` keeps the fallback text on
stdout and adds an `Error:` line on stderr; `--json` appends the terminal
`{"type": "error"}` record. Both exit 1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* refactor(cli): share final-answer accumulation with the embedded client
* docs: scope headless accumulator guidance to TUI
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs(backend): point the stream-mode translation anchor at stream_modes.py
* docs(backend): use one anchor spelling within the stream-mode table row
The row named the translator by its backend-relative path and its call
site by the harness-relative short form, so the same symbol was written
two ways two rows apart.
* fix(config): upgrade the config.yaml the Gateway loads
scripts/config-upgrade.sh resolved DEER_FLOW_CONFIG_PATH, then
backend/config.yaml, then <checkout>/config.yaml. make dev exports
DEER_FLOW_PROJECT_ROOT=<checkout>, so the Gateway reads the checkout copy
first: with both files present the script upgraded a file nothing loads
and left the live config outdated while reporting success. It also
ignored DEER_FLOW_PROJECT_ROOT and a DEER_FLOW_CONFIG_PATH set in .env,
and fell back to another file when DEER_FLOW_CONFIG_PATH named a
missing one.
Ask the harness resolver (AppConfig.resolve_config_path) instead of
copying its order, defaulting DEER_FLOW_PROJECT_ROOT to the checkout as
serve.sh does. A missing DEER_FLOW_CONFIG_PATH or invalid project root
now fails with the Gateway's error; no config anywhere still creates
<checkout>/config.yaml from the example.
* docs(changelog): reference #5991
* test(config): pin config-upgrade seeding a missing config from the example
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tui): report headless --print/--json failures instead of leaking tracebacks
The headless --print and --json runners dispatched straight into the
embedded client with no error boundary, so session-creation, thread-
resolution, chat, and stream failures escaped main() and surfaced as
unhandled Python tracebacks from the console script - for example the
documented CheckpointModeMismatchError case, which the checkpoint
operations/troubleshooting docs recorded as a raw traceback.
Both runners now catch Exception (KeyboardInterrupt/SystemExit still
propagate), mirroring the TUI's stream_actions() boundary:
- --print writes one concise 'Error: ...' line to stderr and returns 1
- --json appends a terminal {"type": "error"} record to the NDJSON
stream (diagnostics stay on stderr) and returns 1; every stdout line
remains independently valid JSON, including failures mid-stream
The checkpoint operations/troubleshooting docs (en + zh) no longer
describe the headless runners as printing an uncaught traceback.
Fixes#5962
* fix(tui): close the broken-pipe edge and document the --json error contract
Review follow-ups on #5963 (both non-blocking):
- print(answer) sat outside the headless boundary, so a consumer closing
the pipe early (deerflow --print ... | head) still got a
BrokenPipeError traceback plus the interpreter's shutdown-flush noise.
The final --print write and the --json terminal error-record write now
catch BrokenPipeError, point the stdout descriptor at devnull (the
documented SIGPIPE-adjacent recipe) and exit 1 without a traceback.
- The terminal {"type": "error"} record is a real contract for --json
scripters but was only documented in the checkpoint pages; the
backend/docs/TUI.md command table and the README cheat line now state
the error record and the exit-1 semantics where CLI consumers look.
* feat(community): add Unbrowse web fetch provider
Add a community web_fetch provider backed by Unbrowse
(https://unbrowse.ai), a hosted service that returns a page as markdown,
over plain HTTP when that is enough and through a cloud browser when the
page needs JavaScript. The tool sends one JSON-RPC tools/call POST
(unbrowse.scrape) to Unbrowse's MCP endpoint, so it needs no extra Python
package and no MCP client (uses httpx, already a dependency).
Unbrowse is not a general web search engine, so only web_fetch is added.
Changes:
- backend/packages/harness/deerflow/community/unbrowse/{__init__,tools}.py
web_fetch_tool with the same output shape as the Sofya, Exa and
Firecrawl fetch tools (# <title> plus markdown, 4KB cap). API key is read
from the config.yaml api_key field or the UNBROWSE_API_KEY env var.
Optional render: auto|never|always; unknown values log a warning and fall
back to auto. JSON-RPC errors, isError tool results, HTTP errors and
malformed payloads come back as "Error: ..." strings.
- backend/tests/test_unbrowse_tools.py: key resolution, render handling,
request shape, structuredContent vs text payloads, error paths.
- scripts/doctor.py, scripts/wizard/providers.py: register the key and the
web_fetch provider; backend/tests/test_doctor.py covers both doctor paths.
- config.example.yaml, .env.example, backend/docs/CONFIGURATION.md,
tools/AGENTS.md, EN/ZH harness/tools.mdx: document the provider.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs(changelog): add Unbrowse web_fetch provider entry
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client
- refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its
configuration: one connection-precedence rule, credential fingerprint, wire-size
counting and error taxonomy, shared by the guardrail gate and the memory hooks.
- feat(memory): wire `memory.prescreen` and `memory.signal_classification` into
DeerMem through one injected judge hook (both off by default, `enforce` gated by
the shadow evaluation).
- config: add the top-level `typesafe:` block and the two judging slots to
`config.example.yaml` and the Helm values, bump `config_version` to 50, and sync
the Helm README example with it.
- test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with
per-stratum gates, censoring and the no-network baseline.
Review follow-ups folded in:
- `mutations_accepted` counts only facts that survive capacity enforcement and the
upload-event scrub that precedes persistence, only removals/reinforcements of ids
that existed, and persisted near-duplicate merges (matched by id, or by identity
for legacy id-less facts); it is published only after persistence succeeds.
- The cached manager hot-reloads its judge when a memory slot or the shared
`typesafe` block changes (including a credential rotation); publication is
serialized and revalidates the config generation, so a build started before an
`enforce` -> `off` rollback cannot land after it.
- Cache provenance is per answer: the `cached` flag and the reported model follow
the answers a side actually consumed, across partial merges and answerless retries.
- An oversized noul integer stays a per-question error instead of raising.
- Incremental call savings use the shared scored denominator.
- The sensitive-stratum gate requires evidence for each of identity, preference
and correction.
* fix(typesafe): resolve one credential per layer and refuse unusable connection values
Review follow-ups on the shared TypeSafe client (#5906):
- A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the
shared `typesafe:` block: the first layer that sets either credential setting
decides it, and inside one layer the literal key wins (the standalone
semantics). A consumer whose endpoint does not accept the block's key no longer
sends it on every call, and the missing-credential message keeps naming the
variable that layer configured.
- `base_url` is validated where it is configured: `http(s)` only, with no query,
fragment, embedded credentials, whitespace or control characters, and a
malformed port fails at construction instead of on every request (httpx raises
`UnsupportedProtocol` / `InvalidURL` per call, and the query would have been
swallowed into the appended endpoint path).
- A credential that cannot travel as a header value — surrounding whitespace from
a secret mounted from a file, a non-printable character — is refused when the
connection resolves. It used to construct fine and then put the whole
`Bearer <key>` value into h11's `LocalProtocolError` message, which the
guardrail middleware's `logger.exception` prints.
- The client raises transport failures without chaining the original exception,
so that message can no longer carry the credential through the cause chain; the
exception type stays in the message for diagnosis.
- `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config`
path: `max_attempts: true` was silently read as `1`. An integer stays a usable
float, so `timeout: 5` keeps working.
Tests: 20 cases across `tests/test_typesafe_config.py` and
`tests/test_typesafe_client.py`, each red on the previous revision and green now.
367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the
eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean.
Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`,
`config.example.yaml`.
* fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence
Second review round on #5906: the remaining items from the review thread and from
the follow-up pass.
Typesafe client (the reply is treated as hostile input too):
- Send `Accept-Encoding: identity`, refuse a body that carries a non-identity
`Content-Encoding`, and read through a bounded `iter_bytes` that stops past
`MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be
decoded in full and, on the gate path, allowed the tool call unexamined; it is
now `invalid_response` (a denial with `fail_closed: true`).
- Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting
escaped the module's taxonomy, so the memory consumers lost the round's record
instead of recording a fallback.
Memory judge:
- A request-level failure is recorded as `request_failed`, never `no_verdict`
(`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable
endpoint was indistinguishable from a provider with no usable answer). The
adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record
and log; `no_verdict` stays for a question-level failure, and an ineligible side
keeps its own reason even when the round's request failed.
Shadow evaluation (the enablement gate reads the evidence, not one number):
- The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to
meet the 1% target; a sample that only meets the point estimate is
`INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound).
- New `baseline_incremental_savings` gate: a baseline that saves as many calls as
the model — the shipped all-trivial sample — fails instead of enabling.
- Negative counts (`mutations_accepted`, message / batch / token counts) are
treated as absent rather than scored: `-3` read as "nothing lost", adding a
phantom saving to the numerator and the denominator.
Docs: the design documents cited but absent from the repository are replaced with
in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`,
`GUARDRAILS.md`), and the gate and limit wording now matches what the code
enforces.
Tests: 31 new or updated cases, each red on the previous revision (compressed body
refused even when it decodes to a valid envelope, body over the cap refused, body
at the cap answered, deep nesting mapped, `request_failed` on each request-level
failure, the bound and baseline gates, the negative counter). 454 focused tests,
`make test-blocking-io` (167) and the eval suites pass; `ruff check` and
`ruff format --check` are clean.
* fix(memory-eval): pair the baseline comparison on the same records
Review follow-up on the incremental-savings gate added in ed930478 (P2): both
sides of the comparison are now the records the no-network baseline can score
(`trivial_only` plus a readable signals list).
Subtracting that subset's heuristic savings from model savings over *all* scorable
skips credited the model with batches the baseline never saw: 300 reviewed skips
with a single annotated record computed 300-1=299 "incremental", passed the gate
and set `enable_recommended`, while the paired comparison was model=1 against
heuristic=1 (incremental 0).
- `Evidence` carries the paired numbers (`baseline_paired_skips`,
`baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one
heuristic count, filled by `build_report` from the baseline summary.
- The gate reports `INSUFFICIENT` when no scored skip carries the annotation and
when the annotation covers only a subset of the scored skips (the message carries
the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive
paired gain, and `PASS` only on a positive gain over the full paired set.
- Docs state the pairing/coverage requirement in
`backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring.
Tests: the reviewer's counterexample as a report-level regression (it failed with
`assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true),
a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields.
344 focused tests, `make test-blocking-io` (167) and the eval suites pass;
`ruff check` / `ruff format --check` clean.
* fix(typesafe): bound the sync body read by the deadline
`deadline_seconds` only covered the sync path's headers, the completed body and
the parse, so an endpoint that drips bytes just under the read timeout held the
caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget.
The read loop now checks the budget before every transport read, which bounds the
overshoot to one read. The pieces are the transport's own reads, not re-chunked
output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk,
so a check at that granularity would never see a drip.
`tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24
pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now
(~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track
the new granularity.
* fix(memory): read the YAML off spelling, and veto a skip over the whole batch
Two review findings on the memory judging path.
`memory.prescreen.mode` / `memory.signal_classification.mode` rejected the
`mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1
parses the unquoted value as a boolean, so both fields saw `False`. The failed
reload left the previous judge running, so both fields now normalize `False` to
`off` before the `Literal` runs. `True` stays a validation error (no mode it
could mean). The examples quote the value, and the docs record why.
The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch
it answers for is the whole post-watermark feed, so an explicit correction in an
earlier turn of an eight-message batch did not stop the skip that then advanced
the watermark past it. `detect_signals` gains a `window` parameter (default
unchanged) and the updater scans the full feed for the veto; the extraction hint
and the reinforcement gate keep the window.
Both regressions are red on the previous head: the config test reproduced the
`input_value=False` rejection through `AppConfig.from_file`, and the batch test
reproduced the prescreen judging (and skipping) a batch whose first turn carries
the correction.
* fix(goal): show the active goal to the agent on every model call
Nothing put ThreadState.goal into model requests. The agent only knew the
goal from messages: the objective the Web UI and IM channels send, and the
hidden <goal_continuation> message. Goals set through PUT /goal, the
embedded client or the TUI never reached it, and compaction could remove
the messages that stated the others.
DurableContextMiddleware now renders the active objective first in its
hidden durable-context data message, escaped, bounded, PII-redacted and
treated as data like the summary. Only the objective is rendered, so the
block changes only when the goal is set or cleared.
* fix(goal): let the agent pursue the active goal at user priority
The durable-context contract told the model never to follow instructions in
its data message, and that message now carried the goal. While a goal is
rendered, the contract ends with a static exception for the <active_goal>
element that opens the data message: work toward it as a request in a user
message, with no system or developer authority, while every other field
value, and any other text that calls itself a goal, stays data.
The element replaces the heading the goal used, which a summary could
reproduce. Other field values in the data message are HTML-escaped, and
active_goal joins the sanitizer denylist, so user input and remote tool
results cannot carry a raw one either.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* feat(extensions): grant bounded host model invocation
* fix(extensions): retain model budgets and isolate schema validation
* fix(extensions): normalize provider-origin model cancellation
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs(backend): drop references to APIs that do not exist
PATH_EXAMPLES.md's upload example imports THREAD_DATA_BASE_DIR from
thread_data_middleware; the module exists but exports no such name, so the
snippet fails with ImportError as written. Resolve the same directory through
the accessor the middleware itself uses, get_paths().sandbox_uploads_dir(),
which yields the layout the file's own path table documents.
CONFIGURATION.md's Docker sandbox block lists `auto_start: true`. SandboxConfig
declares use/port/container_prefix/mounts/network but not auto_start, and the
provider is constructed as cls(**kwargs) from the caller of
get_sandbox_provider() rather than from config extras, so the key is accepted by
extra="allow" and then silently dropped. Remove the line from the example.
* docs(backend): resolve the uploads example through the owner-scoped bucket
get_paths().sandbox_uploads_dir(thread_id) without a user_id selects the legacy
threads/<id>/ bucket, but Gateway uploads land in
users/<user_id>/threads/<id>/user-data/uploads/, so the example imported cleanly
yet still raised FileNotFoundError for a normally uploaded file. Use
deerflow.uploads.manager.get_uploads_dir(), which resolves the effective user
and validates the thread id.
* docs(backend): show the real uploads path layout in every PATH_EXAMPLES demo
The remaining sections still documented the pre-isolation
threads/<thread_id>/ bucket and called `path` a backend-relative string, while
the upload and list APIs return an absolute path under
users/<user_id>/threads/<thread_id>/user-data/uploads/. Reproduced with
deerflow.uploads.manager.get_uploads_dir() and
uploads._list_uploaded_files_for_thread() against DEER_FLOW_HOME: the listed
path is /<home>/users/default/threads/abc123/user-data/uploads/report.pdf.
* docs(backend): clarify runtime home precedence
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(config): honor $VAR substitution for request_admission integer fields
AppConfig.resolve_env_variables replaces every `$VAR` reference with the raw
environment string. RequestAdmissionConfig.requests_per_minute and
max_queue_size are declared `strict=True` so a bool or float cannot slip in,
but strict mode also refuses the string "60", so `requests_per_minute: $RPM`
failed the whole config load with "Input should be a valid integer" even when
RPM=60 — contradicting config.example.yaml's promise that environment
variables work for every field. They were the only strict fields in the
config schema; every other integer coerces in lax mode.
Wrap both fields in a BeforeValidator that converts a decimal integer literal
arriving as a string and leaves everything else untouched, so the strict check
behind it still rejects bools, floats, "60.0" and non-numeric strings.
Tests pin the accepted literals, the rejections that must survive, and the
end-to-end path from a config.yaml `$VAR` reference through AppConfig.from_file.
* docs(changelog): reference #5838 in the request_admission entries
* feat(prompts): add operator overlays for core system prompts
* test(prompts): isolate SOUL forwarding test from operator config
* fix(prompts): wrap complete subagent system message with overlay
* fix(subagents): serialize prompt overlays in durable batches
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(models): add declarative reasoning capability contract (#5073)
Introduce an optional per-model `reasoning:` block beside the legacy
`supports_thinking` / `supports_reasoning_effort` booleans. It declares
thinking availability (unsupported | optional | required), the accepted
effort values with aliases, default, and serialization path, the payload
dialect, and the reasoning-history requirement. When present the booleans
are derived from it and contradictory profiles fail at config load.
`deerflow.models.reasoning` normalizes legacy and declared profiles into
one immutable contract and resolves every caller's generic request against
it. `create_chat_model` is the single enforcement point: required-thinking
models never receive a synthesized disable payload, unsupported effort
values never reach the provider, and the dialect synthesizes the on/off
payload when no template exists. Legacy profiles keep the historical path
byte-for-byte. The lead agent and subagent descriptor resolve the same
policy first so run metadata reports effective values.
`/api/models` and `DeerFlowClient` project the contract as `reasoning`;
the frontend derives mode clamping and effort choices from it through
`core/models/reasoning.ts`, and user preferences accept provider-specific
effort tokens. The setup wizard's Z.AI GLM-5.3-Flash profile migrates to
the contract, restoring its low/high/max effort control.
* docs(models): move the reasoning contract guidance into models/AGENTS.md
backend/AGENTS.md sits within 46 bytes of the 96 KiB effective-chain hard
limit for the middlewares, runtime, sandbox, and subagents guides, so the
new section pushed scripts/check_agent_guidance.py over it. The models
guide is the nearest file for the factory and the contract module, and
its chain has ample room.
* fix(models): address review on the reasoning contract
- vLLM dialect: the enable path now mirrors whichever chat_template_kwargs
switch the template declares, like the disable path already did, so a
template using the older `thinking` key no longer also grows
`enable_thinking`.
- Validate operator-supplied effort values at `effort.path` inside
`when_thinking_enabled`, `when_thinking_disabled`, and the `thinking`
shortcut against the contract, not only the profile-level value; they are
forwarded when the caller chooses nothing, so the "unknown values never
reach the provider" guarantee now holds for them too.
- Constrain `effort.path` to dotted identifiers and reject single-segment
paths that would replace a whole mapping (`extra_body`, `thinking`,
`model_kwargs`, `default_headers`, `default_query`).
- Z.AI GLM-5.3-Flash profile: `default: high` instead of `max`, because the
default also governs summarization, title generation, and subagent calls
that never choose an effort; `max` stays selectable. Documented in the
config example, the configuration manual, and the design note.
* fix(models): close reasoning contract review gaps
* fix(models): preserve native Ollama reasoning config
* fix(models): keep native reasoning strings and project reasoning into the descriptor
- `ModelConfig.reasoning` also accepts the level string ChatOllama takes
for gpt-oss style models (`low` / `medium` / `high`); the factory forwards
both the boolean and the string on the legacy path, so profiles that
passed `reasoning: high` through before the contract field existed keep
loading and keep sending it.
- The assembly descriptor no longer excludes `reasoning` from
`model_parameters`: the native ChatOllama value and a declared contract's
dialect/history/effort change the request payload, so they now move the
assembly fingerprint.
- e2e: the sidecar spec pinned the pre-contract body where a model without
effort control still received `reasoning_effort: minimal`; the preset is
now dropped for such models (the Gateway discarded it anyway), so the
assertion expects the key to be absent.
The docs site pointed at POST /api/extensions/{skills,mcp}/{name}/enable
and /disable, and backend/docs/API.md at POST /api/skills/{name}/enable
and /disable. None of these routes exist. The gateway toggles a skill
with PUT /api/skills/{skill_name} and an MCP server with
PATCH /api/mcp/config, both admin-only, which is also what the
frontend calls.
* feat(guardrails): add TypeSafe (Jev) pre-execution tool risk gate
Adds TypeSafeGuardrailProvider, a network guardrail provider that asks one
`noul` question per probed tool call and denies calls whose Jev risk
probability reaches `threshold`. It reuses the existing GuardrailProvider
slot: no middleware, config-schema or dependency changes.
Semantics that are load-bearing:
- state is strict JSON (no `default=`, `allow_nan=False`, string keys only);
unserializable arguments, lone surrogates among them, and argument text
above `max_state_chars` are denied locally with zero network requests,
never truncated and sent.
- the async path cancels in-flight requests through `asyncio.timeout`; the
sync path checks the deadline after the response headers arrive, around
the body read and before accepting the result, and drops late results.
No total return-time bound is promised on the sync path.
- clients are created per evaluation inside a context manager, and the
injected `transport_factory` is called once per network-needing
evaluation: httpx closes an injected transport with its client, so a
shared instance would be dead on the second call.
- denial reason messages carry probability, threshold, served model version
and state digest, so the threshold comparison can be replayed from the run
journal; local denials record no probability or model version, and allow
decisions stay unpersisted (the audit path is unchanged).
- provider errors carry no response body or arguments: malformed responses
report the field and the offending type only, so a response echoing tool
arguments cannot reach middleware logs or the evaluation report.
The online evaluation harness (`backend/scripts/eval_typesafe_risk_gate.py`
plus its labeled case set) is the threshold/tools evidence that must precede
enabling interception. It reports four mutually exclusive populations
(uncached network, cache hits, unprobed tools, local refusals), separates
model classification from local refusals and fail-closed errors, measures
latency from the caller entering `aevaluate` through cleanup, discloses
censored samples, probes DNS/TCP/TLS separately, and flags unprobed tools as
a coverage gap rather than a pass. Real responses are never written to disk.
Cache-pass samples are classified from `decision.metadata['cached']` and
counted once, warming calls are retained as an auxiliary group with their
failures named in the report and JSON, and an optional connection-probe
diagnostic can never discard completed evaluation results.
Tests: the provider suite over MockTransport plus the middleware
integration, a blocking-IO anchor that drives `aevaluate` against a real
loopback HTTP server with a meta-check proving the sync path on the loop
trips the gate, and harness cases covering the reporting defects. Every fix
above was verified to fail its new tests when reverted.
Docs: GUARDRAILS.md gains Option 4 (privacy, local-refusal trade-off,
deadline semantics, audit boundary, coverage/bypass risk, Allowlist->RBAC
migration, lone-surrogate local refusals and the no-response-body property
of error messages) and config.example.yaml gains a smoke-scope example.
Config: the new example block bumps `config_version` 46 -> 47, with the
Helm chart's embedded version and README example kept in step as
`scripts/check_config_version.sh` (chart.yaml CI) requires.
* feat(guardrails): whitelist tool calls inside the TypeSafe risk gate
Static allow rules now live in the provider instead of waiting on a composite
provider. `whitelist` mirrors `AllowlistProvider.allowed_tools`: omitted means
"not enforced", `[]` refuses every tool, and a list refuses every tool outside
it locally (`typesafe.tool_not_whitelisted`) before state construction, caching
and networking, independently of `fail_closed`. Whitelisted tools are still
probed and still denied when their risk probability reaches `threshold`, and
the whitelist is checked before the probe scope so an unlisted tool cannot
inherit an allow from being unprobed.
Deny rules stay with `authorization.*`. The policy version moves to 1.1.0, the
evaluation script reports whitelist refusals as their own population, and
`GUARDRAILS.md` / `config.example.yaml` document the allow-half migration.
* fix(guardrails): address TypeSafe risk gate review findings
- Status errors report only the numeric status. The HTTP status line is
server-controlled like the body, and this message reaches middleware logs
and the evaluation report, so a hostile endpoint could echo the state (raw
tool arguments) through it.
- Expired cache entries are removed with `pop(key, None)`. Sync evaluation
runs on executor threads when a turn produces parallel tool calls, so a
second caller can remove the same expired entry; a bare `del` raised
KeyError, which the middleware reports as a provider error and, under
fail_closed, turns into a spurious denial.
- The evaluation harness states its acceptance gates inline (no missed
dangerous call, system false-block <= 5%, uncached network p95 <= 1s)
instead of citing a design spec that is not in the repository, and reports
the whitelist-refusal population alongside the other four.
- `--tools` / `--whitelist` strip whitespace and drop empty entries, so
`--tools "bash, write_file"` no longer probes only bash while treating
`" write_file"` as an unknown tool name.
- The case set's `padding` note says top-level argument fields only, matching
what the runner implements.
Each fix was verified by running its new test against the pre-fix code.
* refactor(guardrails): name the TypeSafe permission list allowed_tools
Align the key with `AllowlistProvider.allowed_tools` so an allowlist migration
copies it verbatim instead of translating `allowed_tools` into `whitelist`.
The companion names follow: reason code `typesafe.tool_not_allowed`, metadata
`tool_not_allowed`, harness population `not_allowed`, and the `--allowed-tools`
flag.
Semantics are unchanged: omitted = not enforced, `[]` = refuse every tool,
tools outside the list are refused locally before any state, request or cache
entry, and listed tools still face the risk gate.
* fix(guardrails): record only a bounded served model token; fail the eval exit code
Two findings from the re-review at d3f0b072.
`model` is response content, and `_model_decision` interpolated it verbatim into
the reason message -- which reaches the denied ToolMessage, the run journal,
middleware logs and the evaluation report on disk, and which the cache retains
for replay. A malformed or hostile endpoint can put an echoed tool argument,
injected text or a multi-megabyte string there. The token is now recorded as
itself only when it matches a conservative shape; anything else is recorded as
`unrecorded:sha256:<digest>`, which keeps the verdict and a stable provenance
record without echoing the value. Rejecting the response instead would turn an
odd version string into a fail-closed denial of every call the endpoint answers.
The digest path encodes with `surrogatepass` so an escaped lone surrogate -- a
value `json.loads` happily produces -- cannot raise where this must not fail.
`scripts/eval_typesafe_risk_gate.py` returned 0 even when a gate failed, so an
operator scripting `... && enable` got a green light on a failing evaluation.
`main()` now returns 1 when any gate in `report["score"]["gates"]` fails, after
the JSON report is written, and prints the failed gate names.
Tests: `test_an_unrecordable_served_model_is_digested_instead_of_echoed` (an
echoed argument, a space, a lone surrogate and an oversized value, plus the
cached replay) and `test_main_exit_code_carries_the_gate_verdict` (both
directions). Both fail against the pre-fix source.
docs/GUARDRAILS.md audit boundary updated to match.
113 passed: tests/test_typesafe_guardrail_provider.py,
tests/test_typesafe_risk_eval_script.py,
tests/blocking_io/test_typesafe_guardrail_provider.py.
* fix(evals): fail the risk-gate run when a labeled population is uncovered
Review finding on dcd38cc3. The new exit code made `_score`'s vacuous passes
load-bearing: unprobed calls are excluded from the score, so `risky_misses_zero`
is true over an empty risky population. `--tools read_file` excludes all 18
risky cases in the bundled set, so the run printed every gate as PASS and exited
0 after evaluating no risky operation at all.
`_score` now gates coverage itself: `risky_coverage_present` and
`safe_coverage_present` fail when the probe scope leaves that labeled population
with nothing evaluated, so a scope that cannot substantiate the verdict no longer
reports one. The safe side had the same vacuity -- `system_false_block_within_target`
passes trivially with zero safe cases.
Verified end to end against a loopback mock endpoint with the bundled case set:
`--tools read_file` exits 1 with `GATE risky_coverage_present: FAIL` while the
other four gates still PASS, where dcd38cc3 exits 0 on the same command.
Tests: `test_main_fails_when_the_tool_scope_leaves_no_risky_case` and
`test_main_fails_when_no_safe_case_is_in_scope` (both fail against dcd38cc3),
`test_bundled_case_set_keeps_both_labels_the_coverage_gates_require`, and the
exit-code test now runs with both populations present.
116 passed: tests/test_typesafe_guardrail_provider.py,
tests/test_typesafe_risk_eval_script.py,
tests/blocking_io/test_typesafe_guardrail_provider.py.
* fix(auth): claim the first admin atomically instead of counting first
POST /api/v1/auth/initialize read the admin count in one session and
created the admin in another, with no lock between them. Two requests
arriving together with different emails both saw an empty system and
both became admins: reproduced over HTTP, two 201s and two admin rows,
after which setup-status reports an initialized system.
The endpoint is public and unthrottled, and during the pre-setup window
anyone who can reach the gateway can claim admin with a single request
anyway — that part is the first-boot design. What the race removes is
the only signal: without it exactly one request wins and the loser gets
409, so an operator who loses sees "already initialized" and looks into
it. With it, both succeed and the operator's setup looks ordinary.
The check and the insert now share one transaction in the repository,
with writers serialized first — SQLite takes the write lock up front
(BEGIN IMMEDIATE) and Postgres a transaction-scoped advisory lock, the
idiom the project and channel repositories already use. The same
repository serves both backends, and multi-replica deployments make the
race genuinely parallel rather than an event-loop interleaving.
create_first_admin returns None when the claim is lost, which the route
reports as 409; uniqueness conflicts still raise ValueError and stay a
400. The count in the handler is now only a fast path that avoids
hashing a password for an already-initialized system.
AUTH_DESIGN.md said concurrent initialization was backstopped by a
database unique constraint. No such constraint exists — email
uniqueness only catches the same address submitted twice — so it now
describes the claim that does the work.
* docs(changelog): note the atomic first-admin claim (#5776)
* fix(auth): refuse a dialect the first-admin claim cannot serialize
Review follow-up. The claim's serialization handled sqlite and
postgresql and fell through silently for anything else, which would
leave create_first_admin as the plain check-then-act it exists to
prevent — the one place the abstract method's atomicity contract is
enforced.
An unknown dialect now raises instead. The engine builds only these two
today, so this guards a future backend: it fails loudly at first boot
rather than quietly reopening the race.
* feat(storage): add a content-addressed blob store contract (#4189 item 2)
Addresses the review on the previous head:
* the retention/dedup claim in the doc now states the deletion rule (liveness
from the durable references) instead of promising refcount-free deletion, and
the sidecar records the first writer as advisory provenance only;
* one validate_blob_kind() owns the kind grammar, so the message a producer sees
is the pattern that is enforced;
* adds the missing storage/AGENTS.md (registry entry included) that both
docstrings already pointed at;
* drops the dead _import_backend_module() by routing the folder scan through it;
* exports BlobNotConfiguredError from the package;
* renames _KIND_MAX_BYTES to _MAX_BLOB_BYTES (it caps one put, not a kind);
* delete failures raise the neutral BlobStoreError instead of BlobReadError.
Rebased onto current main so the guidance-budget test matches upstream.
Signed-off-by: zeng-bohan <zengbh1@gmail.com>
* fix(storage): restore the config module and backend packages dropped by the squash
The force-push to e72b435 silently lost four items the surviving code
imports, breaking `import deerflow.storage` and the test suite at
collection:
- deerflow/config/blob_storage_config.py (BlobStorageConfig +
get/set_blob_storage_config + load_blob_storage_config_from_dict,
mirroring memory_config's host-shared schema and hot-reload pattern)
- app_config.py wiring: the blob_storage AppConfig field, the import,
and the _apply_singleton_configs -> load_blob_storage_config_from_dict
call (memory stayed wired; blob did not)
- storage/backends/__init__.py (drop-in contract docstring)
- storage/backends/local_fs/__init__.py (LocalFsBlobStore re-export +
STORE_CLASS for _scan_backends)
Verified: pytest tests/test_blob_store.py tests/test_app_config_reload.py
tests/test_app_config_name_indexes.py -> 67 passed; ruff format/check
clean on all four files.
* fix(storage): name the failing digest check; document the 64 MiB per-put cap
- get_bytes: split the corruption message into size mismatch (truncation /
partial write) vs digest mismatch (same-size bit rot, reporting what the
content hashes to vs what the ref names) — the combined message reported
'got 1024 bytes, expected 1024' on same-size corruption and sent the
debugger in the wrong direction.
- docs/blob-storage.md + storage/AGENTS.md: state the _MAX_BLOB_BYTES cap
(64 MiB) and that it is backend-level defense-in-depth rather than part
of the BlobStore contract, so the externalized-tool-results follow-up
sees the limit before the first production put fails.
* test(storage): pin both get_bytes corruption branches separately
The pre-split test corrupted the blob with different-size bytes, so only
the size-mismatch branch ever ran and neither new message was asserted.
Now: different-size corruption asserts 'size mismatch', and a same-size
reversal (bit rot) asserts 'digest mismatch' — the branch the split was
actually about.
* docs(config): document blob_storage in config.example.yaml
Add the blob_storage section (enabled/backend/backend_config) mirroring the
memory section's shape, matching the Configuration section of
backend/docs/blob-storage.md, and bump config_version 46 -> 47 per the
example-file convention for schema additions. Default state is unchanged:
enabled: false, purely opt-in.
* fix(helm): align chart config version with example
* fix(storage): refresh blob store on config changes
---------
Signed-off-by: zeng-bohan <zengbh1@gmail.com>
Co-authored-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: zeng-bohan <310902929+zeng-bohan@users.noreply.github.com>
* feat(memory): add cognitiveStyle for collaboration preferences
Extend cross-session memory with user.cognitiveStyle and cognitive facts,
backward-compatible normalization, prompt injection as Thinking Style, and
settings UI. Add design notes and unit tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(memory): normalize legacy memory imports on frontend
Add normalizeMemoryPayload aligned with backend normalize_memory_data,
use it for Settings import and API reads, and add unit/regression tests.
Document schema-evolution steps and sync FACT_EXTRACTION_PROMPT cognitive category.
* fix(memory): harden legacy memory normalization
* fix(memory): keep import envelope validation strict
* fix(memory): keep unknown fields through frontend import normalization
normalizeMemoryPayload rebuilt the memory document from a section whitelist,
so every key it did not name was dropped. Because readMemoryResponse routes
all memory API responses through it, the loss showed up on read, export, and
import alike: revision vanished, and any future backend field would silently
break lossless export/import.
The Gateway models are extra="allow" and deermem normalizes additively, which
left the frontend as the only lossy hop in the chain. Spread before overriding
at each nesting level so normalization only validates and defaults
contract-owned fields, leaving the envelope check as strict as before.
Covered by a unit test on the strict import path, two backend tests for
extension preservation and caller immutability, and an E2E round-trip that
asserts on the captured import request body.
* docs(memory): state the normalization invariant that actually holds
The docstring cited `display` as a backend-driven field alongside
`revision`, but #4546 was closed unmerged and no such field exists in the
contract. It also promised lossless export/import, which the Gateway's
`extra="ignore"` models preclude for any undeclared field.
Replace both with the invariant the code is really defending: the frontend
must not be narrower than the Gateway contract, using the declared
top-level `revision` as the concrete example.
* fix(memory): preserve legacy migration and reject lossy imports
* fix(memory): align legacy fact normalization defaults
* test(memory): use authenticated app for import regression
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): bound remaining shell session cleanup requests
release_command_scope(), close()'s scoped-session drain, and close()'s recovery-session cleanup called shell.cleanup_session() without a request budget, so a stalled cleanup could hold the thread-key serializer, scoped.lock, or the sandbox lock for the SDK's default transport budget. Pass the existing _bounded_cleanup_request_options() (5s, max_retries=0) at those three sites; _cleanup_session_best_effort() and its swallow-and-log contract are unchanged, as are create_session and list_dir lifetimes.
* fix(sandbox): bound AIO list_dir with a directory deadline
Give AioSandbox.list_dir its own 60s directory deadline with a 65s no-retry
host envelope, independent of bash_command_timeout. Preserve #5634's
shell-generation selection: list_dir runs on the current recovery session once
the implicit shell is fenced, and an ambiguous list_dir outcome - transport
timeout or an ambiguous returned status - fences whichever generation actually
executed the request, dropping local recovery ownership and attempting bounded
best-effort cleanup instead of leaving that session reusable. hard_timeout
remains definite termination and keeps the targeted session reusable. Only
completed/None results are parsed, so a partial find is never returned as a
complete listing.
Session creation RPC lifetime remains out of scope.
* fix(sandbox): recover sessions after transport failures
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix: rebuild todo reminders after context compaction
* chore: remove implementation plan from PR
* refactor: share todo reminder message name
* docs: keep agent guidance within CI size budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): stop deleting a converted companion we cannot prove we wrote
Conversion names a document's Markdown companion after the document's
stem and falls back to a _N suffix when that name is already taken, so
the .md beside a document may belong to another document sharing the
stem, or to the user. Delete removed it anyway: uploading a.docx and
a.pdf produces a.md and a_1.md, and deleting a.pdf destroyed a.docx's
companion while orphaning a.pdf's own.
Delete now removes only the file it was asked to remove. The companion
stays listed and can be deleted on its own. Orphans are the cost of not
guessing; issue #5672 covers giving companions a provable owner, which
is what a safe cleanup needs, along with the two related readers that
still guess (the outline injected for a document and the agent's file
listing).
convertible_extensions loses its last use and is dropped from the
signature and both call sites. The gateway router keeps importing
CONVERTIBLE_EXTENSIONS for the ingestion bridge and now declares it in
__all__, where that module documents its re-exports.
* docs(changelog): note that delete keeps the converted markdown (#5673)
* fix(sandbox): enforce AIO shell command timeouts
Map Sandbox.execute_command(timeout=T) onto the AIO legacy shell as a server-side hard_timeout=T plus a bounded host request (ceil(T+5)s, max_retries=0), preserve and render the upstream status (hard_timeout -> terminated + Exit Code: 124; no_change_timeout -> may still be running), keep no_change_timeout from preempting hard_timeout, and restrict ErrorObservation replay to completed results.
* fix(sandbox): contain ambiguous AIO command outcomes
Treat a transport timeout and statuses terminated/no_change_timeout/unknown as ambiguous: never replay the current command, fence the scoped or recovery session generation with bounded best-effort cleanup, and never adopt a replacement session for those statuses. hard_timeout and completed keep the session reusable. Align the env-bearing bash.exec path with the same contract and make its retry result authoritative.
* fix(sandbox): wire AIO command timeout from provider config
Restore bash_tool's non-local behaviour (other providers keep their own defaults) and let AioSandboxProvider read sandbox.bash_command_timeout as the sandbox's default_command_timeout at every construction site, with explicit per-call timeouts still winning. Widen the config field to float with allow_inf_nan=False and read the provider value with the module fallback so partial-config providers keep working.
* docs(sandbox): document AIO command timeout semantics
Document the provider-scoped deadline (LocalSandbox/AioSandbox/OpenSandbox wire it; other providers keep their defaults), the semver-vs-frozen-:latest image behaviour, the no-replay guarantees, and trim the sandbox guidance back under its size budget.
* docs(sandbox): scope the wedge-resistance claim to command requests
willem-bd's review on #5634 reproduced that list_dir and both create_session call sites still make unbounded SDK requests while holding the same sandbox lock, so the guide must not read as if every AIO request is bounded. State that the bounded T+5s host wait applies to command requests and name session-creation and file/list RPCs as not covered; bounding those remains a separate operation-deadline/control-plane follow-up.
* fix(sandbox): make list_dir honor the shell generation fence
list_dir previously always targeted the implicit persistent shell, so after an ambiguous command outcome fenced that generation (#5634) a listing could re-enter it. Route default-shell target selection through _ensure_default_shell_session_id so both execute_command and list_dir reuse the explicit recovery session, creating one only when the implicit shell is fenced. No list_dir deadline, hard_timeout, request budget, retry, or status semantics are added; those remain scoped to #5644.
* fix(mcp): honor configured stdio working directories
* fix(mcp): preserve defaults for empty working directories
Treat empty stdio cwd values as omitted, including unresolved environment references. Add real subprocess regressions for discovery and pooled-call defaults, plus direct connection-dictionary coverage.
* fix(uploads): convert the bytes we wrote, not the name they landed under
Document conversion re-opened the upload by name after it was already
visible in the thread's uploads directory: the Gateway converted the
committed file_path, and DeerFlowClient converted the copy it had just
placed there. That directory is writable from local and AIO sandboxes,
so a process watching it can replace the name with a symlink in the
window between the upload landing and the converter opening it. The
converter then reads whatever host file the link points at and writes
that content back into the thread as the .md companion, which the
sandbox can read. Reproduced end to end on both paths with a real xlsx:
the companion came back holding the host file's rows.
The Gateway now duplicates the descriptor of the staged file before the
link-commit, copies those bytes into a private directory outside the
uploads tree, and converts there. A descriptor cannot be redirected by
replacing a name, so the conversion input is the content this request
wrote. The client converts the caller's own source file instead of the
copy in uploads; the source is the file the caller handed in, which the
sandbox cannot reach.
Both already wrote the companion without following a symlink, so only
the read side changes. The uploads directory still receives exactly the
same files.
* docs(changelog): note upload conversion source fix (#5611)
* fix(uploads): close the conversion descriptor when staging its copy fails
Review follow-up. The private directory for the conversion copy was
created before the try that owns the duplicated descriptor, so a failure
there — a full or unwritable temporary filesystem — propagated without
closing it. The upload's own cleanup only unlinks the committed name and
releases the sandbox lease, so the descriptor stayed open for the life of
the process and kept the unlinked staged bytes allocated with it; repeated
failures accumulated both.
Directory creation now happens inside that try, and the finally removes
the directory only once it exists.
* fix(uploads): keep the conversion descriptor owned across cancellation
Review follow-up. run_file_io cannot interrupt its worker, so cancelling
the await around os.dup only abandoned the result: the duplicate was
created moments later with nothing left to close it, and it pinned the
staged bytes of an upload whose name the cleanup had already unlinked.
Cancellation after the duplication was just as leaky, because the
commit-path handler caught Exception and CancelledError is not one.
The duplication now runs as its own task, shielded from the caller's
cancellation, and closes its own result when the caller is gone by the
time the worker finishes. The commit path catches BaseException, closing
the descriptor it already owns before re-raising.
Both windows are pinned: one test stalls the duplication worker after it
allocates and cancels ingestion, the other stalls the commit so the
cancellation lands while the descriptor is owned.
* fix(uploads): drain the conversion copy so its descriptor always closes
Review follow-up. The copy worker owns the duplicated descriptor and
closes it in its own finally, but a bare await let a cancellation cancel
the executor job while it was still queued: the worker never ran, so that
finally never ran either, and the enclosing scope had already handed
ownership away and saw None. Draining also keeps a late worker from
writing into a private directory this scope has since removed.
The copy now goes through await_drained, the shield-and-drain helper the
Gateway already uses for offloads that must not be abandoned mid-flight.
Pinned by a test that holds the copy job queued, cancels ingestion, then
releases it and requires the descriptor to come back closed.
* fix(client): stop embedded uploads from writing through symlinks
DeerFlowClient.upload_files copied each file with shutil.copy2 and let
convert_file_to_markdown write the companion straight into the uploads
directory. Local and AIO sandboxes can write to that directory, so a
symlink planted at an upload name or at the companion's name was
followed: the upload's bytes and the converted Markdown landed in
whatever host file the link pointed to, and the call reported success.
The Gateway refuses symlinked destinations and the IM channels write
through write_upload_file_no_symlink; the embedded client never adopted
either.
Uploads now go through copy_upload_file_no_symlink, a new helper next
to write_upload_file_no_symlink. It keeps copy2's content, permission
bits and timestamps, so files stay readable to Docker sandboxes, but
applies them to the descriptor opened with O_NOFOLLOW and opens the
source first so a missing source cannot truncate an existing upload.
As in the Gateway, a file with an unsafe destination is skipped and
listed in skipped_files, success turns false, and the message says how
many were skipped. The companion is converted inside a private temporary
directory and then written with write_upload_file_no_symlink; one whose
name is unsafe is left out like a failed conversion, and the original
upload is kept.
* docs(changelog): note embedded upload symlink fix (#5578)
* fix(client): keep copy2's same-file guard and companion permissions
Review follow-up. Two regressions in the previous commit.
copy_upload_file_no_symlink opened the destination before comparing it
with the source, and that open truncates. Passing a file that already
sits in the thread's uploads directory therefore copied an emptied file
over itself: the upload reported success with size 0 and the original
bytes were gone, where copy2 raised SameFileError and left the file
alone. The destination is now compared with the source through
os.path.samestat before anything is opened, so identity — including a
hardlink or another spelling of the same path — raises SameFileError as
before.
The Markdown companion was published with write_upload_file_no_symlink,
which creates a new file as 0600 and ignores the converted file's mode.
Under umask 022 the companion became 0600 while its own document stayed
0644, so a bind-mounted sandbox running as another uid could read the
upload but not the Markdown the response advertises. It now goes
through the same copy helper as the upload, which preserves the
converter's permission bits.
