* fix(gateway): reject unusable delegation state at the input boundary
`delegations` is a caller-writable reducer channel. `merge_delegations`
keys on `id`/`run_id`/`dispatch_message_id` and indexes `status`, and the
ledger renderer indexes `description`/`subagent_type` on every model
call, so a non-list value, a non-object entry or a missing field fails
the state merge with an unhandled 500. On the thread-state route the bad
shape is persisted, so every later genuine ledger write keeps failing.
Validate the ledger shape at both caller-writable boundaries and keep
rejecting Overwrite markers and forged verdicts. The reducer also drops
stored values it cannot read, so a checkpoint written before this
validation self-heals instead of failing future writes.
* fix(harness): keep the ledger reducer tolerant of partial entries
The reducer only needs an identity key and a status; the renderer-only
text fields stay a boundary requirement. Dropping entries that lack
description/subagent_type would change reducer semantics for callers
that build minimal entries, so filter only what the reducer reads.
* test(gateway): use valid delegation in provenance check
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): report non-text error results without their payload
When an MCP tool returns isError with no text content, the local conversion
fell back to stringifying the LangChain content blocks. That put the whole
base64 payload into the ToolException message, and the middleware's
500-character cap still left a base64 prefix in the model-visible ToolMessage
and the checkpoint.
Describe the blocks by kind and MIME type instead; text results keep their
existing message unchanged. Applies to stdio tools, which use this conversion;
sse/http tools keep the adapter's conversion (tracked separately).
* fix(mcp): name empty binary error payloads instead of raising
An error result whose embedded file or image had a zero-byte payload hit
create_file_block/create_image_block before the isError branch, so the
model saw ValueError('Must provide one of: url, base64, or file_id')
instead of the server's diagnostic. Skip empty binary payloads on error
results, name their shape in the no-text fallback, and keep normal
results unchanged (audio still raises NotImplementedError).
* feat(frontend): improve file attachment recovery and artifact presentation
* feat(extensions): add Office artifact previews with search and navigation
* fix: preserve public viewers and HTTP Office previews
* fix: address file preview and upload review findings
* fix: use typed upload errors and linear bookmark allocation
* fix(office-preview): isolate parsing and bound preview resources
* fix(ci): install Office preview fixture dependencies for E2E
* fix(ci): provision matching Playwright browsers before E2E tests
* fix(ci): isolate Python libraries and include Office documentation
* test(frontend): wait for download status to finish rendering
* fix(harness): account for tool results in loop detection
* fix(harness): make tool frequency detection observational
* fix(harness): support blocked and waiting plan todos
* style(harness): wrap overlong system prompt lines
* Improve loop detection and todo message formatting
Refactor middleware logic for handling repeated results and update messages for todo list.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Address plan progress and loop detection review
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(sandbox): search user data instead of hiding it as repository noise (#8364)
The ignore list was written to walk source checkouts, but glob, grep and ls ran
it over every root, so /mnt/user-data/uploads hid its own *.log files and a
grep over *.log answered 'No matches found'. grep also skipped any file above
1 MB during a directory walk without saying so, even when the model named the
file itself.
ignore_rules_for() now picks the filter per search root: user-data roots hide
only repository machinery, a glob the caller spelled out wins even inside a
checkout, and a file named as the search root is read at any size. The 1 MB
bound still protects directory walks, and the grep tool documents it.
* fix(sandbox): label user-data roots from the mount table
The public ls/glob/grep tools resolve /mnt/user-data/... to host paths
before they call the local provider, so checking the virtual prefix never
matched a real entry point: an uploaded app.log stayed hidden whenever the
host tree sat inside the checkout the Gateway runs from.
Ask the provider's mount table instead, and keep in-flight .upload-*.part
files hidden in user data as well as in a repository.
* fix(sandbox): keep the repository filter outside user data
Only a root the mount table labels as /mnt/user-data changes behaviour. A
skills mount or an ad-hoc host path keeps hiding *.log as before, so the
existing mounted-root tests stay valid instead of widening content
visibility to every root outside a checkout.
* fix(sandbox): keep an explicit grep budget on a named file
The traversal default stops a directory walk from reading megabytes it was
never asked about. A caller that names one file and passes its own budget
still gets that budget enforced, which the literal-grep suite pins.
---------
Co-authored-by: Aling33 <Aling33@users.noreply.github.com>
* feat(sandbox): add boxd sandbox provider
Adds deerflow.community.boxd_sandbox:BoxdSandboxProvider. Each user/thread
scope runs in an isolated boxd machine. New machines get the thread's files
and skills before they are handed out, release mirrors outputs back to
thread storage, and released machines wait in the shared warm pool.
The boxd SDK is an optional extra (deerflow-harness[boxd]). Unit tests run
against a fake client; tests/test_boxd_sandbox_live.py runs against real
machines when DEER_FLOW_RUN_LIVE_TESTS=1 and BOXD_API_KEY are set.
* fix(boxd): retain failed cleanup and bound output mirror calls
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(search): add DeepSeek web search with configurable thinking
* fix(search): drain DeepSeek teardown and retain citation spans
Reuse the cancellation-safe context helper so repeated cancellation cannot interrupt HTTP transport cleanup. Preserve validated Unicode citation ranges on retained public sources so the agent can associate claims with their supporting pages. Add regressions for cancellation, timeout, normal completion, invalid ranges, URL screening, and result limits.
* fix(search): preserve DeepSeek citation excerpts through guardrails
* fix(search): share DeepSeek citation excerpts across sources
Group each validated answer range once and associate it with response-local source IDs after URL filtering and result limits. Preserve full-answer PII redaction and existing source-card fields, document the model-facing citation mapping, and repair live-test configuration for both thinking modes. Add HTTP transport, guardrail/budget, attribution, and browser reload regressions.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): reject colons in normalize_filename like the skill installer
normalize_filename() accepted a ':' in the basename, so the three upload
writer paths disagreed for the same name on Windows: the streaming write
(write_upload_file_no_symlink) happily stored the bytes in an invisible
NTFS Alternate Data Stream ("notes.txt:hidden" attaches a stream to
notes.txt; directory listings report the base file with 0 new bytes),
while copy/publish raised WinError 123 outright.
skills.installer.is_unsafe_zip_member already rejects any member path
containing ':' with exactly this rationale (drive letters are caught
earlier as absolute paths; a stray colon addresses an ADS that is
invisible to rglob/os.walk listings). Apply the same rule to upload
filenames so every writer path rejects the name identically.
* docs(uploads): clarify colon rejection and drive paths
---------
Co-authored-by: laoyyu <laoyyu@users.noreply.github.com>
Co-authored-by: Nan Gao <thunderggnn@gmail.com>
* fix(agents): run the calls that fit when a batch crosses a per-tool loop limit
One model response with 50 or more calls to the same tool lost every
call: the per-tool frequency check stopped the run before any of them
ran (#8214).
- A per-tool limit now drops only the calls that would reach it; the
calls within the limit run. A response in which no call fits still
stops the run, and the identical-call hard limit still rejects the
whole batch.
- The capped tool stays blocked for the rest of the execution, whatever
the frequency window counts. The cap is a record in the private
loop_cap state channel, so it survives an approval pause, the resume's
new run_id, a Gateway restart and context compaction. before_agent
expires a record another run left behind, which gives a later task a
fresh budget.
- The run is loop_capped from the first dropped call, in the run record
and in a subagent result. A resumed run is marked again from the
record.
- The model is told on its next call how many calls were skipped and how
many were passed on, followed by the wrap-up instructions. The notice
is read from the record on every model call and cleared once the model
has answered.
- The Gateway treats loop_cap as a server-owned state channel.
- middleware:loop_detection gains the action reject_excess.
* fix(agents): count after the window eviction and keep the cap record readable
Review follow-up for #8267:
- A call reaches a per-tool limit when the window would hold that many
calls to its tool after taking the call in and dropping its oldest
entry, as on main. Counting before the eviction rejected a call that
only replaced an older call to the same tool, so 49 reads, one other
call and one more read capped read_file where main admits 49 more.
- The cap record holds only what its reader accepts. A tool whose name
the reader would reject, or one past the record's size bound, is left
out instead of making the next read discard every cap in the record.
The record and its notice are still written, so the drop is explained
and the execution stays loop_capped.
- A dropped batch is audited once per tool that reached its limit in it,
with that tool's own count and threshold.
- Tests: a seeded comparison against a plain sliding window, the
interleaved sequence, unnameable and overflowing records, and the
per-tool audit rows.
* fix(agents): report a partial loop cap when the run ends, not mid-run
Review follow-up for #8267. runtime.context["stop_reason"] is read in
two ways: the worker and Goal continuation read it when the run is over,
while run-input admission, the todo middleware and end-of-graph input
handling read it mid-run as "the run is ending" and skip their checks.
A partial cap wrote it at the first dropped call, while the admitted
calls of that response still had to run, so those calls skipped the
checks:
- input that arrived during inference no longer cancelled the planned
calls, so obsolete writes ran after a steer;
- two parallel write_todos calls both ran, each replacing the plan;
- input that arrived as the run finished was dropped.
A response with no call left is still a terminal stop and is reported
at once. A partial cap is reported to the subagent channel at once and
to the run context by after_agent, which runs after the end-of-graph
input handling; another guard's earlier reason is kept. A run that
pauses for approval no longer reports a reason; the run that resumes it
reports loop_capped when it ends.
Tests drive the three cases through real compiled graphs in the lead
chain's middleware order.
* fix(agents): undo a discarded response's loop accounting and report the guard that stopped the run
Third review follow-up for #8267.
- A response whose admitted calls another middleware discards before any
of them runs (input admission on a steer, approval on calls it cannot
address) no longer leaves a cap or counts toward a limit. As the next
model call starts, the frequency window, warnings and cap record go
back to their state before that response; caps from earlier responses
stay. Before, a steer whose replacement needed the same tool had that
write dropped as already capped. The undo runs in wrap_model_call, not
a before_model hook, so no graph node is added and lead runs do not
spend an extra super-step per turn of their recursion limit.
- A partial cap reports to both channels only when the run ends, and
only when no other guard has given the reason it stopped. A subagent
that a later token-budget stop ended now reports token_capped, as its
run context does, instead of loop_capped.
Tests drive the steer cases through compiled graphs with real input
admission, including a replacement that uses the capped tool and an
earlier cap that must survive the undo, and the token-budget case sync
and async.
---------
Co-authored-by: AochenShen99 <sac1063467113@hotmail.com>
* fix(runtime): report a failed scheduled-goal cleanup on the run stream
The finalizer flips a finished run to `error` when it cannot remove the
occurrence's scheduled goal, but it published no `error` event. The stream
then ended as a success while the record and `/wait` reported the failure:
the web UI raises its run-failure toast and streaming IM channels mark a
turn failed only on that event.
Set the same deferred `terminal_error_frame` the unpersisted delivery
receipt uses, so both finalizer-side terminal errors end the stream the
way the record does.
* fix(runtime): publish a staged error frame only for an error outcome
An interrupt accepted while scheduled-goal cleanup runs keeps the run slot
active. If cleanup then raises, the error is staged, but the cancellation
branch persists `interrupted`; the stale frame made a cancelled run show a
failure toast while the record and `/wait` reported interruption.
Publish `terminal_error_frame` only when the committed outcome is the
error it describes, which also covers a cancellation that wins after the
delivery receipt staged its frame.
The delta cases now set the worker's checkpoint mode, and a new test
cancels the run from inside a failing cleanup.
* fix(runtime): publish a staged error frame only for the committed error
A rollback also commits `RunStatus.error`, so the status check alone still
sent the staged cleanup message while the row and `/wait` said `Rolled
back by user`. Require the frame's message to equal the committed error.
The cancellation test now covers interrupt and rollback over both run
stores and both checkpoint modes; the rollback cases fail without the
message check.
* feat(extensions): recover opt-in workflows unattended after restart
Workflow launches from an authenticated browser session may opt in to
unattended recovery. The launch captures the session's existing authority
ceiling into a bounded, revocable grant on the workflow record; after a
Gateway restart the single-Gateway SQLite store resumes recoverable
records holding a usable grant through the existing journal-replay path,
without a sign-in.
Host: restore_delegated_authority (sibling of restore_batch_authority)
plus capture_authority on bound handles; the per-operation chokepoint
runs an extension-supplied grant validator so expiry and revocation fail
closed on every operation, including artifact reads. Clustered stores
and uncertain admissions stay on the manual path.
* fix(extensions): address review findings on unattended workflow recovery
- grant_validator validates the current persisted grant during locked
admission, where the entry is not published yet, and switches to the
live entry after publication; the previous entries-only lookup 403'd
every unattended restore of a perfectly usable grant
- delegated restore waits for the AgentRuns host installation signal
instead of racing the rest of Gateway startup and failing the scan
- cancellation defers grant revocation to the supervisor's post-cleanup
save, keeping the delegated handle authorized to cancel children the
recovered workflow already admitted
- the per-operation delegated guard rechecks the current account
disablement, which token_version does not reflect
- restore stub revalidates per operation like the real host, with
regression tests for all of the above; format the new fixture and
condense the extension guidance within its budget
* fix(tests): cancel the recovered workflow on its own service
The fixture's action handlers are bound to the original service, which
stop() already made unavailable; the recovered record lives on the
replacement instance, so the cancel must go through replacement.cancel.
* fix(extensions): let a cancelled replayed child keep its cancelled outcome
The replay fence forced every errored re-dispatched step to
needs_reconciliation, including the deterministic case where the error
is the workflow stopping after its child was already cancelled and
observed. Fence only genuinely uncertain outcomes; a clean cancel of a
recovered workflow now ends cancelled like a fresh one.
* fix(skills): reject unextractable archive member trees
An archive with a file that is also an ancestor directory, or a member
that normalizes to the extraction root, crashed extraction with OSError
and the scan preflight raised IndexError on an empty member name, so the
install route returned 500 instead of a validation 400. Validate the
whole member tree before writing anything, comparing NFC+casefolded keys
so case-insensitive and normalization-sensitive filesystems cannot create
the same collision, and enforce the export path/depth/segment limits so
overlong paths fail as validation errors instead of OS errors.
* fix(skills): align archive depth limit with export
export.py limits paths to 32 components relative to the skill root; an
archive member adds the enclosing skill directory, so a valid 32-component
export was rejected on reinstall. Allow 33 archive components and cover the
boundary with a round-trip extraction test.
* fix(skills): reject folded duplicates and enforce the host path limit
Two case-variant file entries collapse into one file on a case-insensitive
filesystem, and a long member path can still exceed the destination
filesystem limit once the staging root is prefixed, raising ENAMETOOLONG
after a partial write. Reject duplicate folded file keys and validate the
full destination path before extraction.
* fix(skills): measure host paths in platform units and guard publication
Win32 limits are counted in UTF-16 code units and long paths are opt-in,
while the previous check compared UTF-8 bytes against 260. Measure with the
platform units and read LongPathsEnabled. Also reject a publication staging
path that exceeds the host limit and translate ENAMETOOLONG during staging
into the installer validation error.
* fix(skills): reject nested members whose staged path is too long
The staging temp-dir name lengthens every member path, so a nested member can
exceed the filesystem path limit while the skill root and the final path fit.
shutil.copytree wraps a child's ENAMETOOLONG in shutil.Error, which carries no
errno or winerror, so the existing translation let it escape as a 500. Check
each member's full staging path before copying.
* feat(opensandbox): configurable sandbox resource limits and requests
The OpenSandbox provider passes no resources to SandboxSync.create(), so
every sandbox gets the SDK default of 1 CPU / 2Gi and the server's default
requests. Workloads that need more (builds, PDF rendering, data work) can't
get it, and clusters with tight quotas can't ask for less.
Add two optional sandbox options, `resource` (limits) and
`resource_requests`, each a map of resource names to quantities, forwarded
to SandboxSync.create(). Numbers become strings; other value types fail at
provider startup. Unset, null or empty keeps today's defaults.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(opensandbox): reject blank or non-positive resource quantities; document requests
Review follow-up for #8305:
- Reject blank or whitespace resource names and quantities, and numeric
quantities that are zero, negative or non-finite, at provider startup
instead of failing at acquire().
- Document that only Kubernetes runtimes apply resource_requests, that an
omitted request defaults to its limit, and that requests must fit within
the limits, including the SDK default 1 CPU / 2Gi when resource is unset.
- Bump config_version to 67 (example, chart values, chart README) and add a
CHANGELOG entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore: drop the CHANGELOG entries (generated in batch by maintainers)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(extensions): pass a task's final status, error, goal verdict and run metadata to on_task_stop
A lead run's final status is decided after the graph has ended: a model
that fails after retries ends the graph normally, so callback-based
tracing reports the run as OK. on_task_stop runs after the status is
persisted but only received completed | failed | aborted.
Additive, so existing contributors keep working:
- TaskInfo.metadata: a read-only copy of the run's metadata (scheduled
runs carry scheduled_task_id, scheduled_task_run_id, scheduled_trigger);
empty for subagents.
- TaskInfo.stop: a new TaskStopDetail (status, error, goal_verdict), set
only on the info passed to on_task_stop.
- deerflow-extension-api 0.2.13.
Refs #8303
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(subagents): compare stop identity without the stop detail; assert its status
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(extensions): keep TaskInfo identity and serialization; flag unconfirmed stop status
Review follow-up for #8304:
- Exclude TaskInfo.stop and .metadata from equality and hashing
(compare=False), so the stop info finds state keyed by the start info.
- Replace MappingProxyType with ReadOnlyDict, a dict subclass that rejects
mutation and freezes nested mappings (lists become tuples). asdict(),
deepcopy, pickle and json.dumps work again; observers sharing one
snapshot can no longer change nested values for the next observer.
- TaskStopDetail.persisted: False when the run store did not confirm the
terminal write or the worker lost ownership, so a staged status is not
reported as final.
- Document waiting_for_input as a resumable pause (outcome failed) and the
persisted flag (en/zh observers and reference).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Nan Gao <thunderggnn@gmail.com>
* fix(storage): anchor the blob-kind pattern at end of string, not before a newline
In Python, "$" also matches just before a trailing newline, so the blob
kind pattern accepted "tool\n" and 64 chars + "\n" (65 bytes) even
though the contract grammar is 1-64 chars and the kind is used as a
filesystem path segment by the local_fs backend. Anchor with "\Z"
instead.
* fix(storage): report the enforced blob-kind pattern
---------
Co-authored-by: laoyyu <laoyyu@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scheduler): fence late launch bookkeeping by claim generation
An expired launch claim can be requeued and immediately reclaimed by the same scheduler process. Because requeue_claimed_run and fail_launching_run only matched status == launching and the lease owner, the original attempt's late busy-thread requeue or launch failure could still clear the fresh claim's lease and release its global execution-budget slot. Another queued occurrence could then claim that slot while the fresh launch was still in flight.
attempt_count already increments on every claim. Capture it from claim_queued_run and pass it to both slot-releasing writes, which now match on the generation as well. The lease-owner check remains; a stale attempt becomes a no-op, while the current generation can still release its own claim. The fence is optional so legacy/custom repositories that do not expose attempt_count keep the previous behavior.
Tests: a SQLite/Postgres repository test recovers an expired claim, reclaims it with the same owner, then replays the stale requeue and failure and asserts the fresh row plus its global slot survive. Service tests pin that the claim attempt is passed on both the busy-thread and pre-launch-failure paths.
* fix(scheduler): log lost requeue claims after a newer generation wins
The busy-thread requeue path discarded requeue_claimed_run()'s boolean, so a generation-guard rejection was indistinguishable from a normal requeue in logs while the attempt no longer owned the occurrence. Capture the result and emit the same lost-launch-claim warning used by the failure path; add a regression test that simulates a newer claim winning.
* docs(scheduler): keep persistence guidance within its soft limit
Compress the generation-fence note so the shared persistence AGENTS.md chain stays below the soft size limit.
* fix(scheduler): omit generation kwarg for legacy claim repos
Both late-bookkeeping calls passed expected_attempt_count=None even when the claim had no generation. Custom repositories that still implement the pre-fence signatures reject that keyword with TypeError. Build the kwargs conditionally so the argument is only sent when the claim actually carries a generation; add a legacy repo regression for both the busy-requeue and pre-launch-failure paths.
---------
Co-authored-by: Tong-bit-art <Tong-bit-art@users.noreply.github.com>
* fix(threads): reject a state update as_node LangGraph cannot compile
POST /threads/{id}/state passes the client's as_node to the state
mutation graph as its only node. LangGraph refuses __start__, __end__,
its other reserved names and names containing ':' or '|', and that
compile runs before the route's error handling, so those values ended in
an unhandled 500.
Check the name in the request model with the same rules, so it is a 422
on the as_node field before any graph is built or state is written. A
parity test keeps the check aligned with LangGraph's own compile.
* docs(runtime): note the LangGraph floor the reserved-name check was verified on
* docs(checkpoints): re-anchor checkpoint_state rows and list validate_state_mutation_node
The new function shifts the cstate line ranges the reference page cites
for CheckpointStateAccessor.bind and build_state_mutation_graph.
* fix(capabilities): build the business client off the MCP event loop
`BusinessClient._request` constructed `httpx.AsyncClient()` inline whenever no
client was injected, which is every production call: `build_server` mints a
client per tool invocation for the bundled dingtalk, wecom and HubSpot tools.
`httpx.AsyncClient()` runs `ssl.create_default_context(cafile=certifi...)` in
its constructor, so each notification and CRM request loaded the CA bundle on
the event loop it was serving.
Construct it through `asyncio.to_thread`, the rule the community fetch clients
already follow (crawl4ai, browserless, groundroute), and add the per-client
`tests/blocking_io` regression the family carries so the inline construction
cannot come back unnoticed.
* Enhance cancellation safety for HTTP client in business.py
Refactor HTTP client construction for cancellation safety and ensure proper closure.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(runtime): publish the delivery failure on the run stream
* docs(api): note the delivery-failure stream event
* test(runtime): cover the deferred delivery-failure frame
* docs(runtime): log the delivery fix and explain the fallback exemption
* test(e2e): present the recorded output in the replay scenario
The write/read-file fixture predates the delivery check, so the replayed
run has been ending as an error without presenting note.txt. This PR makes
that failure visible on the stream, which broke the replay golden and the
render e2e. Add the present_files step the current scenario produces and
regenerate the golden.
* docs: drop the CHANGELOG entry
CONTRIBUTING.md asks contributors to leave CHANGELOG.md and CHANGELOG_zh.md
to the maintainers at merge time, so the Unreleased ordering and the
English/Chinese pair stay in sync.
* fix(scheduler): preserve terminal occurrence when a newer run is active
reconcile_launched_run revived a terminal occurrence whose launch returned late, assuming no other occurrence could own the active slot. A queue timeout or pause can terminalize the old occurrence while a newer occurrence is admitted and active; reviving the old row then violates uq_scheduled_task_run_active with an IntegrityError and aborts the launch path, leaving the durable run unassociated.
Under the parent and occurrence locks, detect another active occurrence first. If one exists, keep the terminal status, error and finished_at, attach only the durable run_id/started_at, and log a warning. Revive to running only when the slot is still free. Tests: repo regression reproduces the SQLite UNIQUE violation on main and passes here; full scheduler suite 1115 passed / 132 skipped; ruff and guidance clean.
* test(scheduler): pin terminal payload preservation in reconciliation
Seed the terminal occurrence with its real error and finished_at, then assert reconciliation preserves status, error and finished_at while attaching the late durable run. A future edit that clears either field now fails the regression.
* fix(scheduler): avoid backfilling start after a terminal finish
The preserve path attached the late durable run's start to an occurrence that had already finished, producing started_at > finished_at. Leave started_at untouched: the occurrence never started, and the durable run row carries the real start. Parametrize the regression over newer occurrences in running and queued status so both active-index shapes stay covered.
---------
Co-authored-by: Tong-bit-art <Tong-bit-art@users.noreply.github.com>
* fix(workspace): treat hidden output paths as working state
get_changed_output_paths counted every regular file under outputs as a
produced artifact, so a finished run that left a cursor/state JSON or a .work/
scratch tree there was recorded as delivery incomplete. Skip paths with a
hidden segment below outputs; visible files still count. Part of #8239.
* fix(workspace): share the outputs prefix and document the hidden-path rule
The delivery filter partitioned on a hardcoded /outputs/ while the scanner
built the same prefix in recorder.py, so a future prefix change would silently
stop filtering working state. Export one OUTPUTS_VIRTUAL_PREFIX used by both,
and record the hidden-path exclusion in the runtime guidance contract.
* fix(sandbox): end the host-path tail at a comma like the other list separators
The reverse-masking tail stops at whitespace and at " ; & | < > ( ) :",
but not at ",". A ,-joined list of host paths in command output therefore
lets the first match's tail swallow every later entry, and those paths
reach the model unmasked -- the same failure mode fixed for :-joined
lists ($PATH, $PYTHONPATH) earlier. The forward patterns have the mirror
defect: a ,-joined list of virtual paths resolves only the first entry.
"," also joins paths in real output (pytest/mypy/sed-style lists), and a
"," inside one path only shortens the match, so no path can be corrupted
by terminating the tail there.
* docs(sandbox): extend the tail-terminator rationale to the comma
Review nits: the _PATH_TAIL rationale comment now explains why `,` ends
the tail alongside `:` (a comma-joined argument listing would otherwise
hide later entries inside the first match's tail), and the snapshot
test's prose no longer calls the `:` stop the only tail change -- the
docstring and the literal's comment both name the `,` delta.
---------
Co-authored-by: laoyyu <laoyyu@users.noreply.github.com>
* fix(gateway): reject an unusable promoted record at the input boundary
* fix(gateway): refuse an Overwrite-marker promotion record
* test(gateway): pin the server-owned metadata strip on promoted
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): read the run and authorization config off the event loop
get_app_config() stats and hashes config.yaml and extensions_config.json
on every call and reparses them after an edit. Run admission read it
inline on the loop two to four times per start (get_run_context,
build_run_config, the model allowlist, the approval switch, conversation
references), and the model list, model details, skill list, suggestions
and sandbox gates read the authorization section inline once each.
start_run now builds its run context in a worker and reuses that
context's app_config for the rest of the start, so one run start sees
one snapshot. Scheduled dispatch resolves its recursion limit in a
worker, run inputs take only the AgentRuns capability, and the
route-scoped authorization helpers run through asyncio.to_thread, as
/api/models/resolve already did.
* docs(changelog): reference #8137 for the off-loop run and authorization config reads
* fix(gateway): validate the tool approval mode against the run's config snapshot
The approval-mode check added on main read get_app_config() inline in
start_run, so every run start revalidated config.yaml on the event loop
again. It now uses the snapshot the run context was built with, like the
model allowlist and the approval switch.
* test(gateway): seed the skill catalog the authorization anchor lists
The skill listing anchor relied on the checkout's skills/public tree
containing academic-paper-review. It now writes one allowed and one
hidden skill under its own skills.path, so it passes without the repo
catalog and still fails if the policy is not applied.
* fix(web-fetch): stop sending .env.example placeholder keys to providers
make config and make setup copy .env.example into .env, and the template
set nine search/fetch provider keys to example values such as
JINA_API_KEY=your-jina-api-key. The default keyless Jina Reader sent
that text as a Bearer key, so every web_fetch on a fresh install failed
with 401 while make doctor reported web fetch as configured.
- .env.example comments those nine lines out, like every other credential
in the template already is.
- provider_credentials.resolve_api_key reads a your-...-key/-token/-secret
value as unset, so existing .env files that still hold the placeholders
recover without editing, for every provider that resolves keys there.
* chore: leave the changelog entry to merge time
Per CONTRIBUTING.md (#8202), maintainers add changelog entries when they
merge a pull request.
* fix(web-search): route every provider key and doctor through one placeholder rule
Review follow-up for #8206:
- provider_credentials exposes is_template_placeholder(), the single
definition of an .env.example example value. key/token/secret must be
a whole hyphen-separated word, so a real value such as your-monkey or
your-passkey is no longer dropped.
- Serper, Serply, Sofya, SpecStream, Unbrowse, Webz, Brave, Tencent WSA
and You.com resolve config and environment keys through
resolve_api_key instead of private strip-and-send copies, so an old
.env that still holds the template value no longer sends it.
- make doctor asks the same predicate: a placeholder key is reported as
still holding the .env.example value instead of "set", and a literal
placeholder in config.yaml falls back to the environment key as the
runtime does.
- A census test keeps credential environment reads under community/
wrapped in resolve_api_key.
* fix(web-search): match shipped placeholders exactly and mirror Browserless precedence in doctor
Second review follow-up for #8206:
- is_template_placeholder() matches exactly the your-... values that
.env.example ships instead of a pattern, so a token an operator chose
(a self-hosted Browserless TOKEN such as your-review-team-token) is
sent again. A test keeps the set equal to the template.
- make doctor mirrors Browserless: a configured token field is
authoritative, blank or not, and BROWSERLESS_TOKEN is read only when
the field is omitted. A blank or placeholder token now warns instead
of reporting the variable, cross-checked against the client.
- The keyless Jina and You.com providers stay ok but name JINA_API_KEY
or YDC_API_KEY when a placeholder there is ignored, and a literal
placeholder api_key in config.yaml is named when no env key backs it.
* fix(doctor): match each web provider's key precedence; drop the dead Jina api_key from docs
Found while re-checking #8206 against every provider's runtime:
- Browserless used as web_fetch had no doctor key check; it now gets the
same authoritative token handling as web_capture.
- InfoQuest reads only INFOQUEST_API_KEY, so doctor no longer counts an
api_key field in its tool config as configured.
- GroundRoute needs a key for every call but had no doctor entry, so a
missing key reported ok.
- You.com names a placeholder api_key in config that it ignores.
- The Jina web_fetch docs snippet carried api_key: $JINA_API_KEY, which
the tool never reads. With JINA_API_KEY now commented out in
.env.example, that line made config loading fail on a fresh install.
- The provider_credentials docstring now scopes doctor's use of the
predicate to web provider keys, and the scripts guide no longer says
.env.example ships live placeholder keys.
* fix(sandbox): match local grep lines without their terminator
find_grep_matches ran the regex on each line with its trailing "\n", so
patterns that can consume the newline (\s$, [^a-z]$, \n) matched every
newline-ended line while the returned line was shown stripped. Strip the
LF or CRLF terminator before matching, as grep(1) and the remote sandbox
providers do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(sandbox): clarify local grep CRLF handling
* test(sandbox): cover literal grep line terminators
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(search): find conversations and jump to matching messages
* test(extensions): align example registry with search contribution
* test(search): scope bookmark search to its extension
* fix(search): address review correctness and scaling gaps
* fix(gateway): reject unusable tool_artifacts entries at the input boundary
merge_tool_artifacts keys every registry entry by handle, so a caller-supplied
entry with a missing or unhashable handle raised while the state update
merged; the thread-state route persists reducer values with Overwrite, so the
bad shape was stored and every later artifact capture failed on it. Validate
the list-of-objects shape with a non-empty string handle on both boundaries,
and skip such an entry in the reducer so a poisoned checkpoint heals.
* fix(gateway): keep the per-entry metadata strip for tool_artifacts
The dedicated registry branch replaced the generic list branch and silently
dropped its _strip_external_metadata_from_message_like pass. Registry entries
are rich dicts, so a caller-crafted entry could persist server-owned message
metadata into the checkpointed registry. Apply the strip to each validated
entry; it is a no-op for legitimate registry entries.
* fix(scheduler): isolate queued run drain failures
* fix(scheduler): isolate queue timeout rows
* fix(scheduler): enforce queue timeout at launch claim
Keep expired queued occurrences ineligible to start when the timeout sweep fails. Cover per-row and whole-scan cleanup failures with SQLite service tests that also verify healthy queued and due work still launches.
* fix(scheduler): exclude expired rows from queue selection
Filter rows past the queue deadline before owner ranking and the fair batch limit so failed timeout cleanup cannot starve healthy waiting work. Keep the atomic claim cutoff as the final launch guard.
---------
Co-authored-by: AochenShen99 <sac1063467113@hotmail.com>
Co-authored-by: Nan Gao <thunderggnn@gmail.com>
* feat(goal): add /goal from-chat for IM channels
First slice of #7050: add an explicit IM command that composes the active goal
from the recent conversation instead of taking the text literally.
Natural-language intent detection remains out of scope.
Gateway:
- POST /api/threads/{thread_id}/goal/from-chat synthesizes the objective
from the thread summary and recent agreement turns (visible user/assistant
text, clarification questions, and card answers), preserving agreed
constraints and the stopping condition.
- Reject a busy thread before the model call; the save reuses the existing
checkpoint-write reservation so a run that starts mid-synthesis still wins.
- Refuse to synthesize while a human-input request is unanswered, and return
structured no_context/unclear/synthesis_failed outcomes without echoing
model-generated reasons.
IM:
- /goal from-chat [hint] shows the saved objective, then starts exactly one
run through the ordinary chat path; a longer read timeout covers synthesis.
- Map busy/pending/unclear/no-context/timeout outcomes to localized replies;
a saved goal whose run cannot start is reported honestly.
- The TUI explains that from-chat is not supported yet; the web input box
still treats it literally (documented follow-up).
Docs: README, CHANGELOG, IM channel guide, TUI guide, AGENTS, and the
acceptance plan/execution record. Tests: new route, synthesis and manager
suites, plus parser and TUI coverage.
* fix(goal): keep the from-chat confirmation visible on every channel
The saved-objective reply was published as a non-final progress update.
QQ and DingTalk card mode drop non-final outbounds before a stream exists,
and Feishu/Telegram fold the update into the run's stream, so the run
output overwrote the objective. Publish it as the turn's final message.
Redact PII on whole lines and the summary before the context budget can
split an identifier; the composed prompt is still redacted once more as
defense in depth, matching the goal evaluator's direct call.
Also correct the gateway/IM docs (from-chat never creates a checkpoint,
the busy pre-check is worker-local, and synthesis uses the deployment
default model), fix the route test clock patch, and add regression tests
for the final-outbound and truncation-boundary cases.
* fix(goal): align from-chat with the goal-route admission gates
Reject a from-chat save while a tool approval is pending, matching
PUT/DELETE /goal, and suppress subagent-batch continuations queued under
the previous goal after the save. Both invariants landed upstream while
this branch was in review; the new route now shares them.
Add route tests for both behaviors.
* docs(changelog): remove the from-chat entry to avoid conflicts
The maintainer asked to drop the changelog entry because the section conflicts with upstream changes. The feature remains documented in the goals guide and README.
* fix(goal): close three review gaps in /goal from-chat
- Enforce model-use authorization for the deployment default model before
the synthesis call, matching the suggestions route, so a denied role
never sends conversation context to the model.
- Save with expected_checkpoint_id pinned to the head the synthesis read;
a write that landed during the model call is a 409 goal_changed instead
of silently clobbering the newer goal.
- Re-check the conversation binding before starting the run: a /new during
synthesis keeps the goal in the original thread and reports that the
task did not start instead of running it in the switched conversation.
* fix(goal): preserve from-chat handoff and run status
Pin the saved thread through run preparation and reject a changed binding.
Read configuration and model authorization off the event loop, and report
rejected starts separately from requests whose response was lost.
Cover the handoff races and streaming/wait admission outcomes through the
real channel manager and SDK transport, plus file-backed config under the
strict blocking-I/O detector.
Co-authored-by: Loveca <44801706+Loveca@users.noreply.github.com>
* fix(goal): preserve reasoning tags in synthesis JSON
Use the existing JSON-aware reasoning stripper so literal tag text in
objective and reason values survives parsing. Cover complete and unclosed
tags while still removing model reasoning before the JSON payload.
Co-authored-by: Loveca <44801706+Loveca@users.noreply.github.com>
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Totoro <msy626836554@gmail.com>
task_tool and batch_task_tool read the subagent registry synchronously
inside their async tool bodies. Every registry read sweeps the
managed-definition store for its cache signature on a one-second TTL
miss - the file store globs and stats the definitions directory, the
SQL store runs a per-row query - and the sandbox-policy check may read
config files. On the streaming event loop that stalls every concurrent
run for the duration of the IO, the same class of loop stall removed
for tool assembly in issue #5172. Dispatch these hops through
run_assembly like the other blocking resolves in the same tools.
* fix(managed-subagents): make signature() change when an update lands in one clock tick
SqlManagedSubagentStore.signature() returns the tuple of (id, updated_at), so
a definition change is only visible to peer processes when the row's stamp
moves. The ORM onupdate callback reads the wall clock, which has ~15.6 ms
resolution on Windows and can also sit behind a stamp written by a node with
a skewed clock. In both cases the new value equals or precedes the stored one
and the registry snapshot is never invalidated.
Force updated_at strictly forward in update(): normalise SQLite's naive
datetime to UTC and, when the clock has not advanced past the stored value,
bump it by one microsecond. Add a regression test that freezes both clocks to
the stored timestamp; it fails without the fix (signature unchanged) and
passes with it.
Refs #8035
* fix(managed-subagents): serialize signature timestamp updates
---------
Co-authored-by: wangkangmao <wangkangmao@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(memory): support category capacity reservation in DeerMem hybrid-v1 eviction
In long-running personal assistant interactions, core user profile facts
(such as health allergies, coding style, or identity constraints) can suffer
from capacity starvation as high-confidence conversational context accumulates.
This change introduces opt-in category capacity reservation for DeerMem:
- In DeerMemConfig and select_facts_for_capacity, support eviction_reserved_categories
(defaulting to ['correction'] for historical compatibility) and
eviction_decay_exempt_categories (defaulting to empty list).
- Wire MemoryUpdater to pass the configured categories to select_facts_for_capacity.
- Add deterministic offline unit tests in test_memory_personal_assistant_protection.py
verifying root-cause reproduction, bidirectional lifecycle (clean explicit removal
and contradiction replacement), and strict backward compatibility under defaults.
* fix(memory): preserve correction priority in reserved capacity
Co-authored-by: Wistom Zhang <77424093+Yakyou7@users.noreply.github.com>
---------
Co-authored-by: Totoro <msy626836554@gmail.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(subagents): keep a steered subagent report in the parent snapshot
A Steer frees the parent's turn before its subagent finishes: wait_tool
returns a 'result is pending' task result and the real report arrives
later as a hidden subagent-result HumanMessage.
ParentContextSnapshot only kept genuine user messages, so it dropped that
report and kept the pending placeholder. A later context_mode=snapshot
delegation therefore inherited a pending result and could not use quotes
the parent had already received.
Keep the hidden subagent-result message as historical conversation data.
* fix(subagents): share the delayed result message name
---------
Co-authored-by: 2SinceJuly <202121119+2SinceJuly@users.noreply.github.com>
Co-authored-by: AochenShen99 <sac1063467113@hotmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
read_batch_item trusted list_items(offset=position) to land on position.
That only holds while item rows are dense: OFFSET-based paging skips
surviving rows over any gap (e.g. a future retention delete), so the
reader would serve the next item's report attributed to the requested
item key, and the tool layer would project next_position from the wrong
base. The browser projection (read_browser_batch_item) already verifies
items[0].position; align the model-facing reader with it and add the
same negative-position guard. Reading past a gap now returns not_found
instead of a shifted row, matching the browser path's semantics.
* fix(agents): count each delegation once when a provider reuses its call id
Tool-call ids are unique per response, not per conversation. Some providers
number every response's calls from call_0, so two task calls in one run can
share an id.
The delegation ledger used (run_id, tool_call_id) as a delegation's identity:
- extract_delegations built one id -> entry map over the whole run, so a
later call_0 replaced the earlier delegation's description and took its
result while it was still running.
- merge_delegations replaced the same (run_id, id), keeping one entry.
- SubagentLimitMiddleware counted distinct ids, so the per-run cap
(subagents.max_total_per_run) never advanced past the first call_0 and a
run could launch any number of subagents one response at a time.
Each entry now also records dispatch_message_id, the AIMessage that issued the
call, and results pair through pair_tool_call_results, which answers a call
only from its own response. The reducer, DurableContext's change filter and
the cap use the extended identity. An entry without a dispatch id still
updates its run's entry for that call id, and a dispatch-scoped entry adopts a
ledger entry written before the field existed, so runs in flight across the
upgrade are not counted twice. The steer path names the dispatching AIMessage
from the tool request's state, so its delegation update and the graph's own
capture stay one entry.
* fix(agents): close an earlier run's dangling dispatch by its own response
_close_delegations_left_by_earlier_runs decided "answered" by
(run_id, tool_call_id). When a stopped run had dispatched call_0 twice,
the first reply kept the second, dangling entry in_progress, so the
lead kept hearing "already delegated" for work nothing would finish.
Entries that carry dispatch_message_id are now decided by pairing that
response's calls through pair_tool_call_results. Legacy entries and
dispatches no longer in history keep the id-based inference.
Also pin that two runs reusing call_0 stay two ledger entries, and
cross-reference the three delegation identity helpers.
* fix(agents): settle an approval-gated task delegation in the ledger
With `task` in `hitl_approval.tools`, the paused run records the dispatch
as in_progress and the resume run executes it. The resume run opens with
no HumanMessage, so its capture window starts after the pre-existing
messages and never contains the dispatching AIMessage: the result was
dropped and the entry stayed in_progress with no result_brief, forever.
DurableContext now pairs results the resumed window receives with the
last pre-existing dispatch and settles the issuing run's entry, keeping
its run_id instead of re-tagging it. HITL denials of `task` carry
`cancelled` subagent status, so a rejected or denied delegation settles
too.
* fix(agents): address review on the delegation ledger
- Steer path: hold each wait's dispatch AIMessage locally and pass it to
the observer, so a later response reusing the call id while the first
call is still observed cannot take over its args or dispatch id.
- New-turn close: a dispatch whose reply carries subagent status settles
from that reply instead of staying in_progress, covering a crash
between the tools checkpoint and the next capture.
- Reuse thread_state.delegation_key instead of a duplicate identity
helper in DurableContext.
- CHANGELOG: reference #8128.
* docs(changelog): drop the #8128 entry per review
* fix(memory): reject duplicate fact ids in imports regardless of entry order
A backup containing two entries for the same fact id was accepted or
rejected depending only on their order. The duplicate check in
_commit_changes_locked consulted the map of prepared writes, but an
entry whose normalized content matches the stored fact is skipped from
that map - so an unchanged entry followed by an edited duplicate passed
the check while the reverse order raised, and the saved snapshot ended
up with whichever entry came last.
Track every incoming id in a seen-set for duplicate detection, before
the unchanged-entry skip. The saved snapshot now survives the rejection
in both orders.
Fixes#8150
* fix(memory): validate duplicate import ids before capacity selection
Copilot review follow-up on #8153: the storage-level duplicate guard runs
after _select_for_capacity, and the capacity policy can drop one of two
same-id entries (e.g. lower-confidence copies trimmed to max_facts) - the
duplicate then never reaches the guard and the import outcome still
depends on entry order.
Validate all normalized incoming ids in import_memory_data before
capacity selection. The storage-level guard stays as defense in depth.
Regression test with max_facts=10 and 9 distinct high-confidence facts
plus two same-id copies.
* fix(memory): surface duplicate fact id errors with their own detail
willem-bd review on #8153, three follow-ups:
1. P2: the bare ValueError fell through to the router's generic catch, so
POST /api/memory/import answered with the facts-array message even
when every fact had non-empty content. Add
MemoryDuplicateFactIdError (same type as #8151) at the manager
boundary and map it to its own 400 detail that names the duplicated
id; router test asserts the exact detail.
2. P3: the duplicate check lived inside the apply_changes-override
branch, so the base MemoryStorage save() path bypassed both guards.
Move it before the branch split, right after normalize_memory_data,
matching #8151's validate-before-capacity posture.
3. P3: parametrize the order-dependence regression over the default
bucket (agent_name=None) and "researcher", and assert the seeded
snapshot survives the rejection in both orders.
* test(memory): pin the storage duplicate guard and fix the no-op assertion
willem-bd review follow-ups on #8153:
1. The snapshot-survival assertion compared storage.load against
get_memory_data, which is itself storage.load - both reads happened
after the rejected import, so a regression that partially applied the
duplicate would pass. Capture the seeded snapshot before the rejected
import and compare against it.
2. The updater gate rejects import duplicates before storage is touched,
so nothing exercised the storage-level guard in _commit_changes_locked
with an unchanged duplicate first - reverting it would keep the suite
green. Add a direct apply_changes regression test; verified it fails
with the guard removed.
* style(memory): apply ruff fixes after #8151 merge resolution
* test(memory): assert untouched store in the storage-guard pin; facade import
willem-bd review follow-ups on #8153:
1. The storage-guard regression test pinned rejection but not the
untouched-store half: capture the seeded snapshot right after seeding
and assert storage.load still equals it after the rejection, so a
regression that partially applies the first entry before raising
fails.
2. The router regression test imported MemoryDuplicateFactIdError from
the manager module directly; use the package-facade import consistent
with the other manager-error usage (the symbol is already re-exported
from deerflow.agents.memory since #8151).
* test(memory): drop the mock router duplicate-id test
ShenAC-SAC review on #8153: #8151 already drives the real import route
with duplicate IDs and asserts the same 400 response plus an unchanged
snapshot, and the mock test's docstring implied a detail the assertion
did not check. The direct apply_changes() regression is kept.
Also removes the blank line after the docstring that tripped
ruff format --check on the repo-pinned Ruff version.
* test(memory): remove the orphaned mock-router test body
ShenAC-SAC review: my earlier deletion removed the function definition
and docstring but left the test body running inside
test_clear_route_keeps_working_for_windows_reserved_agent_name. Remove
the whole block and the now-unused MemoryDuplicateFactIdError import;
#8151 already covers the duplicate-ID response through the real import
route.
* fix(subagents): apply tool policies to middleware-declared tools
* fix(subagents): address tool-policy review feedback
Normalize declaration conversion errors with middleware context, document
generated discovery exemptions, and add the Unreleased changelog entry.
Audit final explicit and middleware tool sets before graph creation, with
identity-based exemptions for generated discovery helpers and regressions.
* fix(subagents): reuse tool name resolution and clarify policy errors
Server-prefixed tool names are not unique across servers: server 'a'
exposing 'b_x' and server 'a_b' exposing 'x' both bind as 'a_b_x'. The
flatten step forwarded both declarations to the model, where the provider
silently keeps only one and the other server's tool becomes unreachable
with no diagnostic.
Deduplicate by bound name in enabled-server declaration order and drop
later duplicates with a warning.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>