* feat(memory): integrate FTS5 retrieval adapter
* deps: add jieba as default dependency for Chinese tokenization
Without jieba, FTS5 unicode61 tokenizer treats entire Chinese sentences
as single tokens, making single-character or sub-phrase searches
impossible (e.g. '吃' or '油泼面' returns 0 hits against
'用户喜欢吃油泼面'). jieba segments Chinese text into meaningful tokens
before indexing.
* fix(memory): avoid treating hyphens as FTS5 operators
* feat(memory): make Chinese tokenization optional
* fix(memory): warm every requested retrieval scope
* fix(memory): close retrieval resources on shutdown
* fix(memory): close backend when shutdown flush fails
* fix(memory): recreate corrupt retrieval index
* fix(memory): tolerate partial retrieval rebuilds
* fix(memory): warm retrieval index in background
* fix(memory): preserve shutdown flush budget
* fix(memory): stop retrying partial lazy rebuilds
* fix(memory): close retrieval through storage
* refactor(memory): simplify retrieval scope limit
* docs(memory): clarify retrieval shutdown lifecycle
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): share inbound webhook dedupe across pods via Postgres (#4120)
* ci: run cross-pod inbound dedupe integration tests in CI
Expose the job Postgres service via DEDUPE_TEST_POSTGRES_URL so the integration tests (issue #4120) actually execute instead of silently skipping. Normalize the URL for asyncpg (postgresql:// -> +asyncpg, drop libpq-only sslmode) and await the now-async _is_duplicate_inbound in test_github_dispatcher.
* feat(sandbox): add Tenki cloud sandbox provider
Adds deerflow.community.tenki, a SandboxProvider backed by Tenki cloud
microVMs, alongside the existing e2b_sandbox / boxlite / aio_sandbox
backends. Selected via `sandbox.use: deerflow.community.tenki:TenkiSandboxProvider`
(resolved by class path, so the change is purely additive).
The full Sandbox contract is implemented — execute_command plus
read/write/update/download_file and list_dir/glob/grep — with file ops run
as busybox-portable shell commands (cat / find / grep / chunked base64),
reusing deerflow.sandbox.search, mirroring e2b_sandbox and boxlite. Tenki's
SDK is synchronous, so unlike boxlite there is no event-loop bridge.
Tenki sandboxes run as an unprivileged user with /mnt root-owned, so the
/mnt/user-data virtual prefix is remapped under the writable home dir (like
e2b_sandbox); the provider also best-effort sudo-symlinks /mnt/user-data to
that home dir so agent shell commands using the literal path still work.
Sandboxes are pooled per (user, thread) with warm reclaim, a replica cap,
and an idle reaper via the shared WarmPoolLifecycleMixin. Transient
transport blips get one bounded retry; terminal session errors evict and
recreate.
Only the stable Tenki surface is used (create/terminate + exec/shell/fs) —
no volumes, snapshots, or template builds — so any stock base image works.
The tenki-sandbox SDK is an optional extra (deerflow-harness[tenki]) and is
imported lazily, so a default install and every other provider are
unaffected.
Tested: unit suite runs in CI without tenki-sandbox installed; a live
integration test and full-surface e2e were verified against real Tenki
sandboxes.
* fix(sandbox): remove unsafe auto-retry from Tenki exec
Pre-merge review caught that the transient-transport retry sat at the
universal _exec layer, so it retried every operation — execute_command and
base64 file-write chunks included. gRPC has no exactly-once guarantee: a
"socket closed" ack-drop after the server already ran the op means the retry
runs it twice, double-firing command side effects and duplicating a write
chunk mid-file (silent binary corruption on multi-chunk writes).
exec is not idempotent, so it must not be auto-retried. Reverts to the
boxlite/e2b behavior: a transient error surfaces to the caller (returned as
text by execute_command, raised by the file ops); a terminal session error
still evicts the sandbox so the next acquire rebuilds it. Verified live
end-to-end across 31 edge cases (empty/binary/unicode/chunk-boundary files,
shell-metachar content, error paths, list/glob/grep, warm-pool reclaim,
concurrency).
* fix(sandbox): address Tenki provider review feedback
- Use Tenki's native sandbox.fs API for all file transport (read_text,
read_bytes, write_stream, mkdir) instead of cat/chunked-base64 over shell.
Uploads stream in 1 MiB frames; append is read-modify-write because the
write stream has no append mode (same approach as community/e2b_sandbox).
- download_file streams via fs.read_stream and enforces the 100 MB cap on
bytes actually received, closing the TOCTOU window between the old
wc -c size probe and the read.
- list_dir/glob/grep report paths back under /mnt/user-data instead of the
sandbox-internal home dir, so results feed straight into the file APIs.
- Create with wait=False and await wait_ready() here: create(wait=True)
raises with the session handle still inside the SDK, leaking a running
microVM this provider could never terminate.
- Configure the sandbox lifetime (max_duration, default 4h) and expose
sticky; without it Tenki reaps a reused thread's sandbox after ~30 min.
- close() terminates before marking the adapter closed and re-raises real
failures, so a failed termination stays retryable instead of silently
leaking a billed microVM; an already-gone session still counts as closed.
- Bump the optional extra to tenki-sandbox>=0.4.0 and commit backend/uv.lock.
* fix(sandbox): scope tenki grep() glob filter to its directory prefix
Mirrors #4168, which fixed the same defect in the E2B provider. The tenki
adapter reduced a directory-scoped pattern like "src/*.js" to its basename
before filtering, so the search silently broadened to every matching-extension
file in the tree. Post-filter grep's hits through path_matches() against the
path relative to the search root, the same way glob() already does, so both
agree on what a directory-scoped pattern means.
* fix(sandbox): address Tenki provider review — eviction, id width, write lock, grep -H
Four fixes from the upstream review:
download_file no longer swallows terminal transport errors. The broad
`except OSError: raise` re-raised ConnectionError/BrokenPipeError/EOFError
(all OSError subclasses that _is_terminal_failure treats as terminal) before
_note_failure ran, so a session that died mid-download was never evicted. Only
our own EFBIG size-cap now passes through without eviction.
Sandbox id widened from 32 to 64 bits (`[:8]` to `[:16]`), matching
community/e2b_sandbox. The warm pool is keyed by this id with no full-seed
fallback, so a collision could let one user reclaim another's parked sandbox on
a multi-tenant gateway.
_fs_op now holds the lock across the op, not just the fs lookup, so concurrent
calls on the same sandbox serialise over the SDK's shared connection. The
eviction callback runs after the lock is released to avoid a lock-order
deadlock with the provider. The append read-modify-write is serialised by a
dedicated _write_lock so two concurrent appends can't clobber each other.
grep passes -H so a search whose path resolves to a single file still prints
the filename; without it the file:line:text unpack dropped every match.
* fix(sandbox): address Tenki provider review round 2
- validate config `environment` at load time (_validate_extra_env) so a bad
key fails fast instead of surfacing as an SDK error mid-command
- document the deliberate lock decision in download_file: the instance lock is
dropped before streaming so a 100 MB download can't block every other tool;
terminal transport errors still evict via _note_failure
- tighten the terminal-error comment to note ConnectionError/BrokenPipeError/
EOFError are also treated terminal via isinstance
- document TenkiSandboxProvider in backend/AGENTS.md (provider detail, warm-pool
destroy hook, community provider list)
- add a commented Tenki block to config.example.yaml for parity with AIO/BoxLite
- tests: config env validation, grep -F/case-sensitive flags, glob include_dirs,
list_dir max_depth, bootstrap-failure warning branch
* fix(sandbox): make Tenki bootstrap non-interactive and time-bounded
The create-time bootstrap runs under the per-scope acquire lock, so a hang
would stall acquire for that scope indefinitely:
- use `sudo -n` so a password-requiring sudoers entry fails fast (swallowed by
the existing `|| true`) instead of blocking on a tty password prompt
- pass a timeout to the bootstrap `remote.exec` so any other stall drops to the
existing warning path rather than wedging acquire
Best-effort by design; the file APIs still work via the home remap on failure.
* test(sandbox): pin Tenki bootstrap timeout to its actual value
Assert bootstrap["timeout"] == _BOOTSTRAP_TIMEOUT instead of `is not None`, so
a regression to timeout=0 (treated as no timeout by some SDKs) or an unrelated
value is caught rather than passing a weaker non-None check.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(dingtalk): support inbound file and image attachments
DingTalk previously dropped picture and file (document) messages because
`_on_chatbot_message` ignored any message with empty text, so users could
not send files to the agent. This adds inbound attachment support, mirroring
`FeishuChannel`:
- `_extract_files` parses `picture`/`richText` image downloadCodes and `file`
(document) descriptors. `dingtalk_stream.ChatbotMessage.from_dict` does not
parse `file` messages, so `_DingTalkMessageHandler.process` stashes the raw
callback payload on the message (`_df_raw_data`) for the document descriptor.
- `receive_file` downloads each attachment by `downloadCode` via the robot
`messageFiles/download` OpenAPI, persists it into the thread uploads bucket,
syncs it into a non-local sandbox, and prepends the sandbox virtual path to
the message text so the agent can read the file by path.
- Filenames go through the shared `uploads.normalize_filename` helper, which
strips directory components and rejects traversal patterns.
Outbound `send_file` already existed; this completes DingTalk file parity with
Feishu on the inbound side. Adds 21 tests covering extraction, download-by-code,
persistence/sandbox sync, filename sanitization, and the handler raw-data stash.
* fix(dingtalk): address inbound-file review feedback
Follow-up to the review on #4423:
- Make the fallback filename safe by construction. `download_code` is
attacker-controllable webhook data and was embedded into `fallback_name`
unsanitized; it only avoided escaping the uploads directory because the
resulting write failed with OSError. It is now restricted to
`[A-Za-z0-9_-]` before use. Covered by a test that reproduces the old
behaviour (`uploads/dingtalk_../../evil.png`) and by a test that actually
exercises the previously untested `except ValueError` branch (`".."`,
whose basename — unlike `../../etc/passwd` — does raise).
- Log the swallowed `get_image_list()` failure instead of silently returning
no images, so an SDK parse failure is distinguishable from a richText
message that genuinely has no inline images.
- Surface failed downloads to the agent as a short `[failed to load ...]`
marker rather than silently omitting the attachment, so a user whose file
did not load does not simply appear to be ignored. Keeps the cleaner text
shape while restoring the signal Feishu provides.
Tests: 119 passed (was 115).
* fix(dingtalk): claim unique upload names and refuse symlinked destinations
Round 2 review follow-up on #4423. Both findings reproduce as failing tests
against the previous head.
- Inbound attachments no longer overwrite each other. Generated names repeat
across messages (every picture message yields "image.png", richText yields
"image_0.png"), so a later attachment silently replaced an earlier one whose
virtual path had already been prepended to the message text — the agent could
read bytes that were not the ones its prompt referenced. The destination name
is now claimed with the shared `claim_unique_filename` against the live
directory contents, which also covers a real filename sent twice
(`quote.xlsx`), a case Feishu's inline naming does not handle either. The
claim and the write happen under one lock so two attachments cannot resolve
to the same free name.
- Writes go through the shared `write_upload_file_no_symlink` instead of
`Path.write_bytes`. Uploads dirs may be mounted into local sandboxes, so a
sandbox process could leave a symlink at a future upload name and redirect a
gateway-privileged write outside the bucket; the regression test shows the
old code creating the out-of-bucket target.
Tests: 123 passed (was 119).
* fix(dingtalk): harden the inbound download path (self-audit)
Proactive hardening pass over the new inbound path; each fix reproduces as a
failing test against the previous head.
- Contain token failures. `_get_access_token()` sat outside the try in
`_download_by_code`, and the manager awaits `receive_file` without one — a
DingTalk auth hiccup during a file message aborted the whole chat turn with
no reply. Token acquisition moves inside the try, and `receive_file` gains
per-attachment isolation so no unforeseen error can escape past the marker.
- Cap inbound size. The download buffered arbitrary bytes in memory
(`response.content`) with no limit, while outbound uploads already enforce
one. The body is now streamed and dropped once it exceeds
`_MAX_INBOUND_FILE_SIZE_BYTES` (50 MB), surfacing as a failed-load marker.
- Sanitize the failure marker. It embedded the raw webhook `fileName`; a
newline could forge a standalone `/mnt/user-data/uploads/...` line inside
msg.text and an over-long name bloated it. Markers now collapse whitespace
and cap at 80 chars.
- Keep blocking IO off the event loop. `ensure_thread_dirs`, the uploads-dir
resolve, sync `SandboxProvider.acquire`, and `sandbox.update_file` all ran on
the loop; directory prep now lives inside the same `asyncio.to_thread` as the
claim+write, and sandbox sync uses `acquire_async` + an offloaded
`update_file`. Locked by a strict Blockbuster anchor
(tests/blocking_io/test_dingtalk_receive_file.py), verified to fail with
`BlockingError: Blocking call to os.mkdir` when the offload is reverted.
Tests: 127 + 1 blocking-io anchor (was 123); tests/blocking_io/ suite 55 passed.
* fix(dingtalk): surface missing-sandbox sync as a failed load
Round 3 follow-up on #4423:
- When a non-local sandbox acquire succeeds but the provider cannot resolve
the instance, _receive_single_file returned the virtual path anyway — a
path the agent's sandbox cannot read. Mirror Feishu: log and return "",
so the [failed to load ...] marker fires instead. Red-first test:
test_missing_sandbox_after_acquire_yields_marker.
- Drop the dead GetResponse / FakeClient.get scaffolding left in
test_oversized_download_is_dropped from its red-first iteration.
Tests: 128 + 1 blocking-io anchor (was 127 + 1).
* fix(dingtalk): treat non-local sandbox sync failure as a failed load
Round 4 follow-up on #4423. The sync except-branch logged and still returned
the virtual path when acquire or update_file raised on a non-local sandbox —
the same handing-the-agent-an-unreadable-path failure mode the sandbox-is-None
branch was just fixed for, and exactly the leg the suite did not exercise.
Feishu's except-branch returns its failure marker; DingTalk now does the
equivalent (return "" so the failed-load marker fires). Red-first test:
test_update_file_failure_yields_marker.
Tests: 129 + 1 blocking-io anchor (was 128 + 1).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(clarification): structured form fields for human-input cards
Add a request-side v2 `form` mode to the ask_clarification protocol so
business flows (e.g. expense reimbursement) can collect several values
in one card instead of sequential free-text questions:
- `ask_clarification` gains a restricted `fields` parameter (text /
textarea / number / select / multi_select / checkbox / date)
- ClarificationMiddleware validates and normalizes fields explicitly
(whitelisted types, unknown -> text, select-likes without options ->
text, duplicate/invalid entries dropped, all-invalid falls back to
the legacy modes) since the middleware short-circuits before tool
execution; the plain-text fallback lists fields for IM channels
- Form payloads carry `version: 2` so older frontends degrade to the
text fallback; replies stay on the v1 response protocol — the card
submits a readable summary as `response_kind: "text"`, so journal
persistence and answered-card recovery are unchanged
- Frontend renders typed field controls with required-field validation
and compact multi-select chips
Part of #4400 (scope narrowed per maintainer feedback: request-side
only, no new response kinds, no top-level multi_choice).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(clarification): harden form protocol per review feedback
Address the five review points on #4406:
- Reject field names colliding with JS Object.prototype members on both
sides; frontend reads form values via own-property access only, so
`constructor`/`toString`-style names can no longer leak inherited
members into required validation or the submitted summary
- Close open requests answered through the legacy text fallback: a
visible plain human reply (no response metadata) now marks every
previously-opened request as answered, so upgrading to a v2-aware
frontend cannot leave the composer locked on an already-answered card
- Give checkbox fields deterministic boolean semantics: values are
seeded to an explicit false ("no" in the summary) and `required` means
must-agree/consent; documented in the tool schema
- Make middleware field validation atomic: structurally broken entries
(bad/duplicate/reserved names, over-cap field/option counts or text
lengths) degrade the whole form instead of silently dropping fields;
options are trimmed/deduped with blanks removed so the backend never
emits payloads the frontend parser rejects
- Associate form labels/controls (htmlFor/id), aria-required,
aria-invalid, and error descriptions for accessibility
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(clarification): type the fields item schema via TypedDict
Replace `fields: list[dict[str, Any]]` with `list[ClarificationFormField]`
(a TypedDict with `name` required and the type whitelist as a Literal) so
the provider-facing tool schema documents the item shape instead of an
opaque object relying on the docstring. Runtime validation is unchanged
and stays in ClarificationMiddleware, which intercepts the call before
tool execution. Addresses the non-blocking review suggestion on #4406.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): drop unsupported aria-invalid from multi-select group
jsx-a11y: role=group does not support aria-invalid; the error linkage
stays via aria-describedby.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(clarification): coerce numeric required flags and normalize fields once
- `_normalize_bool` now coerces 1/0 (some providers serialize booleans
as integers), so `required: 1` no longer silently flips to optional
- `_handle_clarification` normalizes `fields` once and passes the result
to both the text fallback and the payload builder
Addresses the non-blocking review nits on #4406.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(clarification): harden form protocol per contract review round 2
Backend:
- Guard unhashable JSON in the intercept path: `type: []`/`{}` degrades
the field to text and `clarification_type: []` coerces to str instead
of raising TypeError (which, with return_direct, ended the turn with
an error and no card or fallback)
- Add a total budget over the serialized normalized fields (16KB UTF-8
bytes): per-item caps alone admitted forms whose IM text fallback
exceeded channel delivery limits (Slack 40k chars, Feishu ~30KB card),
silently truncating trailing fields; a boundary test proves any
accepted form's fallback stays deliverable
Frontend:
- Submission value now appends a JSON block keyed by stable field names
(readable summary alone is delimiter-ambiguous), with a collision
regression test
- Parser boundary tightened to match backend constraints: empty option
values (Radix SelectItem crash), duplicate option ids/values,
duplicate field names, and the form<->version-2 binding are rejected
- Keep the error node mounted while any field is still invalid so
aria-describedby never points at a removed element (happy-dom
interaction test)
- Required semantics are now accessible: native checkbox control (no
HTML required attribute — it would intercept the custom submit path),
visually-hidden localized "required" markers next to the aria-hidden
asterisks
- Legacy-fallback closure narrowed to the latest unanswered request:
nothing guarantees a single outstanding clarification across runs, and
closing all would silently swallow older decisions; an older request
left open becomes the active card again
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): keep clarification selects controlled
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(checkpoint): production-shaped full/delta benchmark with configurable snapshot frequency
- Group benchmark scripts into per-family folders (checkpoint/, sandbox/)
- Extract shared benchmark infrastructure into checkpoint_bench_common.py
- Add checkpoint_delta_snapshot_frequency config (default 1000, process-frozen);
freeze it in make_lead_agent and DeerFlowClient; key the state-schema
adaptation cache by resolved frequency
- New bench_production.py: per-case child processes run N ainvoke turns through
the real lead-agent graph (scripted deterministic model, real AsyncSqliteSaver),
then measure GET /state + POST /history through the real Gateway route stack
in one event loop (httpx ASGITransport), cold/warm accessor-cache split,
cross-mode digest gates
- New summarize_production.py: delta/full ratios plus decision metrics
(snapshot_write_spike, cache_effect_ms, checkpoint_write_share,
auto-discovered history per-limit ratios)
* fix(checkpoint): address production benchmark review
* fix(runtime): linearize delta-mode checkpoint resume
Resuming a run from an older checkpoint forks the lineage, and in delta
mode that fork's state cannot be materialized correctly: the delta
history walk collects every pending_writes entry stored on each on-path
ancestor, but a shared parent also carries the writes of the sibling
child that was abandoned. Those writes replay into the fork, so the run
starts from a message list that still contains the answer it was meant to
replace — regenerating in a branched thread surfaced this as the
superseded assistant message reappearing beside the new one after a
reload. All three saver implementations are affected, so write-to-child
ownership is a gap in the upstream delta contract rather than one
saver's slip.
Rather than reimplement that walk, express the fork as what it means:
materialize the requested checkpoint's state, write it as an Overwrite on
the current head (which has no siblings), and run linearly. The abandoned
turn stays in history as the rewritten head's ancestry.
This runs after the rollback point is captured, so cancel-with-rollback
still restores the real pre-run head, and fails closed — an unreadable
resume checkpoint raises instead of falling back to the corrupt fork.
Full mode keeps forking: its checkpoints carry complete channel_values
and need no replay.
* fix(runtime): restore complete delta resume state
* fix(runtime): linearize delta rollback restoration
* Apply suggestions from code review
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(runtime): serialize delta resume preparation
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* feat(memory): signals-based update pipeline + always-on watermark/trivial filter
Refactor the DeerMem memory update pipeline (message_processing -> queue ->
updater) around a signals frozenset seam, replacing the
(filtered, correction_detected, reinforcement_detected) 3-tuple with
(filtered, signals: frozenset[str]) end to end.
message_processing:
- Externalize signal-detection patterns to YAML (message_patterns/*.yaml).
- Extend signals from correction/reinforcement to a 6-class set
(correction/reinforcement/preference/identity/goal/decision); detect_signals
returns a frozenset aligned with the fact category enum.
- Pure-acknowledgment turns ("ok"/"好的"/...) are always filtered out before
enqueue (whole-message fullmatch), saving an extraction LLM call.
queue (core/queue.py):
- In-memory list + debounce timer, with flush_sync (graceful-shutdown drain
that joins an in-flight worker under a hard timeout) and queue_max_depth
backpressure (signal-bearing updates always admitted; QueueFull otherwise).
- Same-key updates coalesce with a signal union; per-batch success/fail summary.
updater (core/updater.py):
- head500+tail500 message truncation (replaces the 1000-char head chop).
- Always-on per-thread watermark: feed only messages added since the last
extraction. The watermark is in-memory and is not advanced on failure, so a
failed/lost update is re-fed on the next conversation turn.
- [MANUAL] prompt marker for user-authored facts (source.type="manual").
- Post-invoke extraction_callback (host-injected) emitting facts_extracted /
facts_accepted / rejected_low_confidence; the host default logs metrics and
flags >60% rejection.
Confidence filtering remains in _apply_updates (the existing
fact_confidence_threshold check); there is no separate write gate.
Consolidation stays opt-in (lossy). The ABC add/add_nowait signature is
unchanged, so the summarization flush hook and host are unaffected.
Tests: add test_message_processing_signals, test_updater_truncation,
test_updater_watermark; update queue/updater/consolidation/staleness/pluggable
tests for the signals seam.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): harden update pipeline per PR review
- Catch QueueFull in DeerMem.add/add_nowait so backpressure degrades to
'update skipped' instead of propagating into after_agent /
summarization_hook and breaking the agent run (peer middlewares
self-guard; MemoryMiddleware was the lone exception). Emergency
(add_nowait) always admits under backpressure -- its data cannot be
re-fed next turn.
- Rewrite the watermark from index-based to content/identity-based
(_message_identity + _feed_after_watermark) so it stays correct when
summarization removes the conversation front -- an index watermark
pointed at the wrong message and silently skipped un-extracted tail
turns. The emergency flush bypasses the watermark (bypass_watermark on
ConversationContext, threaded through update_memory) and coexists with
(does not replace) a pending normal update, so a flush cannot drop a
pending update's un-extracted tail.
- Populate facts_accepted / rejected_low_confidence inside _apply_updates
at the real confidence-filter site (passed_threshold) instead of
re-deriving the threshold in _finalize_update -- eliminates metric drift.
- Emit extraction metrics in a finally with an 'attempted' flag so
exception failures (parse error, apply_changes raise after retry) are
observable, not only the happy path.
- Re-detect signals on the post-watermark feed for the extraction hint so
it no longer references turns the LLM cannot see; admission-time signals
still drive backpressure.
- Move the post-batch reschedule inside the queue lock to close a
non-atomic self._timer race with a concurrent add.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): address follow-up review nits (LRU, metric name, docstring)
- Bound the in-memory watermark cache with a configurable LRU
(watermark_max_keys, default 4096, 0=unbounded). A dropped key re-extracts
one batch on that thread's next turn (the documented restart behavior), so
eviction is safe and preserves the content-identity watermark's
front-removal guarantee. Adds _watermark_get/_watermark_set helpers and a
bounded-LRU regression test.
- Rename the extraction metric facts_accepted -> facts_passed_confidence so
the name matches what the >60% rejection-rate warning assumes (a
confidence-gate signal, not a persisted-fact count); drop the stale
"historical semantics" justification. Brand-new callback, one consumer.
- Fix the stale test_message_processing_signals module docstring: the signals
seam is already swapped to frozenset, and a stale stage-numbering prefix is
removed.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(frontend): pin recent chats
* fix(threads): address pin-chat review feedback
- Stop bumping updated_at on metadata-only PATCH (pin/unpin) via a new
update_metadata(touch=False) path so unpinning no longer jumps a chat
to the top of the updated_at-sorted recent list.
- Narrow patchThreadMetadata to a ThreadMetadataPatchResponse matching
the Gateway's actual response (no values/context).
- Namespace the pinned metadata key as deerflow_pinned for consistency
with deerflow_sidecar / deerflow_branch.
- Cover touch/touch=False behavior in repo + router tests; document the
e2e mock's updated_at preservation now mirrors production.
* style(frontend): format thread utils test
* fix(threads): make pinned ordering server-side
* test(frontend): keep infinite-scroll fixture order stable
* test(frontend): stabilize lark reconnect e2e
* docs: clarify thread pin metadata contract
RunCreateRequest declared stream_resumable as None-only, but langgraph_sdk
defaults it to False (not None) and its payload filter only drops None, so
every SDK request carried "stream_resumable": false and was rejected with 422.
False means "no resumable stream", which is what DeerFlow already serves.
This broke every IM channel run (runs.stream and runs.create both send the
field; only runs.wait omits it) and any external langgraph_sdk client. The web
frontend was unaffected because its compatibility wrapper does not send it.
An explicit true still returns 422. The new regression test drives a real SDK
client to capture its default payload and posts that to the HTTP boundary, so a
future non-None SDK default cannot regress this silently.
Fixes#4466
* fix: surface length-capped model responses
* fix: avoid the influence of the mid-turn
* fix: correcting semantic annotations
* fix: add ModelLengthTerminationDetector to compatible providers
* fix:delete redundancy code
* fix:supplementing log information improves observability
* fix: align the document and complete the assertions.
* fix: unit test
* fix: revert AGENTS.md
* fix: unit test
* fix: add annotation and skip AIMessage has empty content
* fix(gateway): stop persisting Overwrite wrappers into empty reducer channels on branch
Thread branching (and POST /state on a never-written channel) wraps copied
reducer values in Overwrite. Upstream BinaryOperatorAggregate.update seeds
an empty (MISSING) channel with values[0] verbatim without unwrapping, so
Union-typed channels (sandbox/goal/todos/promoted) stored the wrapper
literally and the next consumer crashed with TypeError: 'Overwrite' object
is not subscriptable (#4380). Patch the channel to unwrap the first write
(mirroring DeltaChannel semantics), and stop copying thread-scoped channels
(sandbox, thread_data) into branches: the parent's sandbox_id would bind
the branch to the parent's workspace and release lifecycle.
* refactor(checkpoint): drop private _get_overwrite import for a local Overwrite check
Importing langgraph's underscored _get_overwrite at module top level meant an
upstream refactor that drops it - plausibly the same release that fixes the
bug - would fail this module's import and crash startup before the probe can
stand the patch down. Replace it with a local helper on the public Overwrite
type, and fix two test docstring nits.
* refactor(checkpoint): write patch flags via their constants to avoid drift
Both saver patches read their "already patched" idempotence flag through a
module constant (_PATCH_FLAG / _BINOP_PATCH_FLAG) but wrote it as a hard-coded
attribute literal, so renaming the constant would silently break the guard and
double-apply the patch. Write via the same constant (setattr), dropping the
now-unneeded attr-defined ignores.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): unwrap Overwrite-wrapped sandbox state in after_agent
Fork-restored checkpoints can deliver the sandbox channel still wrapped
in langgraph.types.Overwrite: the rollback restore applies replace-style
writes through a state-mutation graph in delta checkpoint mode, and
after_agent/aafter_agent then crash subscripting the wrapper
("TypeError: 'Overwrite' object is not subscriptable") on the next
sandbox tool run in the forked conversation. Unwrap before reading the
sandbox id, and pin both hooks against an Overwrite-wrapped state.
Refs #4380 (bug 1 of 2; the history-loss half is a separate display path)
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(sandbox): don't release fork-restored parent sandboxes
The Overwrite-wrapped value replays the parent thread's sandbox state, so releasing it from the forked run would evict the parent's warm sandbox. _unwrap_sandbox now reports the wrapped form, and both after_agent hooks skip the release for it while keeping the normal path unchanged.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: add lark cli integration
* fix: polish lark integration actions
* feat: support lark incremental permissions
* fix: detect lark authorization completion
* fix: harden lark integration install
* feat: expand lark auth scopes and reuse host auth in sandbox
Default lark auth to least-privilege (recommend=false, base sign-in only)
and expose the full set of lark-cli --domain business domains as native
--domain grants instead of a 4-domain read-only mapping. Resolve the
skill pack from the latest larksuite/cli GitHub release at install time
with content-hash integrity, and surface version/runtime drift in status.
Share the per-user lark-cli config/data profile between the Gateway
Settings auth flow and agent conversations by mounting the integration
dirs into the AIO sandbox and injecting the matching env for lark-cli
commands, with an allowlisted extra_mounts path in the provisioner/K8s
backend and traversal guards on integration paths.
* style: fix lint issues from ruff and prettier
Sort imports in the provisioner PVC test and re-wrap two long i18n
description strings to satisfy backend ruff and frontend prettier CI.
* fix(lark): address managed integration review feedback
* fix(frontend): stabilize integrations settings e2e
* test(sandbox): isolate remote backend legacy visibility check
* test: fix backend unit failures after merge
* Harden Lark integration review fixes
* Format Lark integration E2E test
* fix(lark): harden sandbox credential exposure and status disclosure
Address willem_bd's security review on PR #3971:
- Mount the per-user lark-cli config dir (long-lived appSecret) read-only
into the AIO sandbox; only the refreshable-token data dir stays writable.
- Redact host filesystem paths (install_path, cli.path) from
GET /lark/status and the config/auth complete responses for non-admin
callers, fail-closed on any auth error.
- Document the npm postinstall trade-off (--ignore-scripts is not viable
because @larksuite/cli fetches its platform binary in postinstall).
- Document the sandbox credential trust boundary in AGENTS.md and README,
pointing at the sidecar-broker follow-up (#4338).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Branch creation seeds the new thread's run-event feed from its checkpoint
so inherited history survives the first run (#4380). Every seeded row
carried one shared run id, but run_id is a *turn* identity to the feed's
consumers, not a provenance tag: regenerating the inherited answer
resolves that row's run id as the superseded source, and
GET /messages/page then drops every row carrying it. One shared id for
the whole seed therefore deleted the complete inherited history on a
branch's first regenerate, leaving only the regenerated turn.
Group seeded rows into one synthetic run per inherited turn
(branch-seed-{thread_id}-{n}), a new turn opening at each persisted human
message — the same boundary a real run has, including the allowlisted
hidden ask_clarification reply, which resumes as its own run. Supersession
is then confined to the turn actually regenerated.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(summarization): own the run model for compaction; bound failure
With summarization.model_name: null the summary model resolved to
config.models[0] while the executing model is selected per run; when they
differ and models[0]'s provider is broken (expired key, quota, outage)
compaction silently failed every triggered turn and context grew unbounded
until the main provider 400s the run (#3103's shape), even though the run's
own model was healthy.
Model ownership is now sourced from the builders, not re-derived at runtime:
- The lead, subagent, and manual /compact builders each pass the resolved run
model into create_summarization_middleware(run_model_name=...). The middleware
no longer reads runtime.context / get_config(), which do not carry a custom
agent's or a subagent's resolved model, so a custom-agent lead run and a
distinct-model subagent now summarize with their own model, not models[0] /
the parent's. Runtime re-resolution and the per-name model cache are removed.
- model_name: null summarizes with the run's own model; an explicitly configured
summary model generates and falls back to the run model on failure. The
fallback is built lazily after the primary fails and its construction is
guarded, so a broken fallback cannot skip a healthy primary or escape the
automatic failure boundary.
Failure is bounded and side-effect-safe:
- An empty or whitespace-only response is treated as a generation failure, not a
valid summary, so compaction never removes all history for an empty replacement.
- compact_state/acompact_state take raise_on_failure independent of force: the
manual /compact path always surfaces a generation failure (even force=false)
and routes it to the existing ContextCompactionFailed path (HTTP 500 ->
frontend error toast) instead of an unconsumed response reason. The automatic
path leaves compaction state unchanged.
- before_summarization hooks fire only after a replacement summary exists.
SummarizationConfig.model_name, config.example.yaml, and docs/summarization.md
document the final lead/subagent/manual ownership rules.
Part of RFC #4346 (section A). Evaluating fraction/triggers against the run
model's profile (profile ownership) is a separate follow-up.
* fix(summarization): manual /compact model ownership + fail-open construct/parse
Manual /compact carried only agent_name, so it derived the run model from the
custom-agent model or config.models[0] and missed the request-selected model the
run path uses (request -> custom-agent -> default). Carry model_name through
ThreadCompactRequest and the frontend compact call, resolve with the same
precedence, and move the custom-agent config read off the event loop (asyncio
.to_thread) with user_id so the strict blocking-IO gate is not bypassed by the
broad except.
Make one summary attempt own its full lifecycle so the fail-open boundary covers
construction and response parsing, not just invocation: build each candidate model
lazily and guarded (a raising constructor falls through to the healthy run model
instead of breaking agent construction), build the model_name:null primary from the
run model rather than config.models[0], and run response text extraction inside the
invocation try so a failing .text accessor falls back instead of escaping compaction.
Adds factory-level constructor-failure, response-extraction-failure (sync/async), and
route-path model-ownership tests.
Queue comments received while a run is active, then submit one deduplicated follow-up after it finishes. Failed drains are requeued and watcher tasks stop cleanly with the channel manager.
* fix(skills): offload blocking filesystem IO in get_custom_skill_history
The GET /api/skills/custom/{name}/history handler ran its storage probes and the
per-skill .history read directly on the asyncio event loop:
get_or_new_skill_storage(), custom_skill_exists(), get_skill_history_file().exists()
and read_history() are all blocking filesystem IO. make detect-blocking-io flagged
the existence probe (routers/skills.py:224) as DIRECT_ASYNC.
Move the whole read into a nested sync function run via asyncio.to_thread; a None
return signals 404 (distinct from an empty history list). Behavior is unchanged.
Per the blocking-io-guard SOP:
- Candidate: get_custom_skill_history (FILE_METADATA, DIRECT_ASYNC) -> FIX+ANCHOR.
- Re-scan: the finding no longer appears for this handler.
- Anchor: tests/blocking_io/test_skills_router.py drives the real handler against a
real on-disk skill + history; teeth verified red (pre-fix) -> green (post-fix)
under make test-blocking-io.
Scoped to this self-contained read handler. rollback_custom_skill and update_skill
also touch blocking IO but interleave it with awaits (security scan / cache refresh)
and do a read-modify-write, so offloading them needs the asyncio.Lock serialization
treatment (cf. #3552) and is left as a separate fix unit.
* test: trim dead skills history setup
* fix(skills): use the user-scoped storage accessor in the offloaded history read
The merge with main left the offloaded reader calling get_or_new_skill_storage,
which is not defined in this module (ruff F821), so lint failed and the handler
would raise NameError at runtime. Use _get_user_skill_storage(config) — the same
accessor every other handler in this router uses.
Also update the regression test for the current route signature: the handler is
now admin-only and takes a Request, so the test supplies request.state.user
(mirroring tests/blocking_io/test_channel_runtime_config_store.py) and seeds the
history through the same user-scoped accessor.
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(subagents): align prompt and middleware subagent limit; allow min of 1
SubagentLimitMiddleware clamped max_concurrent to [2, 4] internally, but
agent.py and client.py fed the raw config value into the system prompt, so
a user-configured 1 (or 5) produced a prompt that disagreed with the
enforced middleware limit. Lower MIN_SUBAGENT_LIMIT to 1 and clamp the raw
config value with _clamp_subagent_limit() at both the agent factory and the
embedded client so the prompt and middleware see the same value.
* fix: remove unused imports MAX_CONCURRENT_SUBAGENT_CALLS, MIN_CONCURRENT_SUBAGENT_CALLS, clamp_subagent_concurrency
* fix: harmonize clamp range [1,4] across middleware, config, and prompt path; fix lint
- Changed MIN_CONCURRENT_SUBAGENT_CALLS from 2 to 1 so prompt.py's
clamp_subagent_concurrency and the middleware's _clamp_subagent_limit
both clamp to [1,4] — eliminating the divergence where the prompt told
the model 'max 2 task calls' but the middleware enforced 1.
- Applied _clamp_subagent_limit at build_middlewares (agent.py:360) so
all 3 construction sites (agent.py:360, agent.py:450, client.py:259)
consistently clamp the config-resolved limit.
- Derived MIN_SUBAGENT_LIMIT / MAX_SUBAGENT_LIMIT from
MIN_CONCURRENT_SUBAGENT_CALLS / MAX_CONCURRENT_SUBAGENT_CALLS so the
two module-level definitions stay in sync.
- Added TestConfigParity.test_prompt_path_and_middleware_clamp_agree
regression test.
- Fixed lint.
* fix(lint): add missing imports for MIN_CONCURRENT_SUBAGENT_CALLS and MAX_CONCURRENT_SUBAGENT_CALLS
* docs+test: update AGENTS.md clamp range to 1-4; add prompt/middleware parity regression test
- backend/AGENTS.md still documented the old [2,4] clamp in two places;
updated to [1,4] to match MIN_CONCURRENT_SUBAGENT_CALLS = 1.
- Added test_apply_prompt_template_single_subagent_limit_matches_middleware:
renders the real system prompt with max_concurrent_subagents=1 and asserts
the advertised HARD LIMITS value equals SubagentLimitMiddleware's enforced
max_concurrent — the end-to-end check that would have caught the [1,4] vs
[2,4] prompt-path divergence flagged in review.
* refactor: simplify per review — restore clamp delegation, drop redundant call-site clamps
Per willem-bd's review, reduce the PR to the one behavioral change plus
docs/tests:
- _clamp_subagent_limit delegates to clamp_subagent_concurrency again
instead of inlining a byte-identical copy; with a single source of
truth the TestConfigParity sync-check class is unnecessary — dropped.
- Revert the call-site clamps in agent.py (build_middlewares,
_make_lead_agent) and client.py (_ensure_agent) to main: both
downstream consumers (SubagentLimitMiddleware.__init__ and the prompt
path) already clamp internally, and the cross-module private import
of _clamp_subagent_limit goes away with them.
- Keep MIN_CONCURRENT_SUBAGENT_CALLS = 1 (the fix), the [1, 4]
docstring updates, the AGENTS.md range corrections, and the
end-to-end prompt/middleware parity test for single-subagent mode
(docstring reworded: on main a configured 1 was bumped to 2 by both
paths — there was no divergence to fix, just a silently raised floor).
* test: fix stale comment referencing reverted agent.py/client.py call-site clamps
---------
Co-authored-by: nankingjing <nankingjing@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Enforce one queued or running scheduled-task run per task with a partial unique index. The migration resolves legacy duplicates before creating the index, and losing inserts use the existing conflict or skip outcomes.
Normalize YAML frontmatter keys in the shared parser so validation and review report malformed fields instead of failing while sorting mixed key types.
Add GIF to the view_image allowlist: map the .gif extension to
image/gif and detect the GIF87a/GIF89a magic bytes so the existing
extension/content cross-check accepts GIFs instead of rejecting them
as an unsupported format. Covered by a new success test.
