* fix(mcp): reconcile committed Gateway config mutations
Install selective MCP binding fences inside the Gateway config commit boundary and finish teardown after releasing config locks. Reject frozen durable-task server changes before persistence.
Preserve drained cancellation semantics and cover delete/re-add, durable-only bindings, multi-server diffs, shared writers, and post-write reconciliation failure.
* test(mcp): pin committed handoff boundaries
Prove the fallback fence is installed before releasing the config lock, a committed writer cannot swallow an earlier shared reset, delete/re-add advances the applied baseline and cache generation, and retries after a post-write 500 have explicit conflict or missing semantics.
* fix(mcp): fence committed config after reload failure
Treat a post-write reload failure as a committed handoff failure so local MCP state is detached before releasing the config lock and teardown remains outside it.
* fix(mcp): always finish committed pool teardown
Run finish_mcp_reconciliation from a post-lock try/finally so response or logging failures cannot strand a retired full-reset pool.
* test(mcp): fence cold-start binding race
Pin the in-flight discovery path that keeps the committed fast path from admitting a late old-fingerprint binding.
* docs(mcp): document committed teardown guarantee
* fix(mcp): sanitize committed response errors
Keep post-write response conversion failures behind a fixed HTTP detail while preserving the retired-pool teardown guarantee.
* fix(mcp): avoid duplicate committed reconciliation error logs
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): record the content actually overwritten in custom-skill history
Scope:
- backend/app/gateway/routers/skills.py: move the prev_content read of
update_custom_skill and the current-content read of rollback_custom_skill
into their drained persist tails (read -> write -> append_history in one
worker thread, no awaits in between). The blocked-rollback branch now
builds its informational entry separately. No behavior contract changes:
status codes, cancellation draining, and prompt-cache refresh order are
untouched.
- backend/tests/test_skills_custom_router.py: two regression tests that
inject a concurrent write during the scan phase and assert the history
entry's prev_content equals the revision actually overwritten on disk.
Why: prev_content was read before the security scans, so a concurrent
revision landing between that read and the persisted write was overwritten
on disk but absent from the history chain, making it unreachable by a later
rollback and desynchronizing the audit record.
Tests: uv run pytest tests/test_skills_custom_router.py --basetemp=... ->
36/36 passed (34 existing + 2 new); ruff check and ruff format --check clean
on both files. Python 3.12.13, Windows 11.
* fix(skills): serialize custom-skill mutations and pin the concurrency regression to the real window
- Add a module-level mutation lock shared by the edit, delete, and rollback
paths, covering the full read -> write -> history-append sequence so a
concurrent PUT cannot land between a request's predecessor read and its
write (previously the projection lock started inside write_custom_skill,
leaving that window open).
- Replace the two regression tests: they now pause the first mutation inside
its storage write after the predecessor read (deterministic gate), complete
a second real HTTP PUT inside that window, and assert the history chain
stays replayable. Verified both tests fail against the base implementation
(2 failed) and pass with the lock (2 passed); full file 36 passed,
ruff check + format clean.
* fix(skills): scope the custom-skill mutation lock to the owning user
The lock serializing the read -> write -> append sequence was one process-wide
lock, and that sequence wraps write_custom_skill's entire user projection
rebuild. A user waiting on their own projection lock, or copying a large or
slow linked package, therefore stalled every other user's edit, rollback, and
delete of unrelated custom skills, even though their skill trees, histories,
and projection roots are disjoint. The base storage only serialized those
mutations per user projection.
Key the lock on the canonical owning-user storage root instead, so one user's
mutations remain a single critical section — including across separately
constructed storage objects, which a hot-reloaded config or a later request
creates — while two users never share it.
The read -> write -> append sequence still cannot interleave within a user, and
the lock is still acquired and released inside the worker thread's own with
block, so caller cancellation cannot strand it.
* fix(skills): serialize custom history across gateway processes
---------
Co-authored-by: sjr666666 <sjr666666@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(brave): contain malformed web search results
* fix(brave): log malformed responses and isolate test transport
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): select the model per conversation via /model
IM channels could list models with /models but had no way to choose one:
the model for a conversation came only from channel/session config or the
server default, so a user who needed a stronger or cheaper model for one
task had to reconfigure the whole channel or move to the web UI.
- /model <name> pins a model to the current conversation, validated
against the caller-visible /api/models list (list membership doubles
as the authorization check), persisted in thread metadata so it
survives a Gateway restart and applies from the next message in
place — unlike /agent use, which must start a new conversation.
- The pin is written to all three runtime carriers (run context,
configurable, context) after the session layers merge, so it wins
over channel/session configured models; like any explicitly
requested model it also takes precedence over a Custom Agent's
configured model.
- /model (no args) reports the effective model and its source through
one resolver that mirrors the run merge order (default -> channel ->
user, context carrier over configurable, both session-config
carriers scanned); /model default clears the pin; /models names the
pinned model in a footer.
- Pin lookups degrade instead of failing messages: the status query
reports "Current model: unknown" on a lookup failure (matching the
/models footer), and the reuse-thread preload leaves the in-process
cache untouched so the next turn retries a transient threads.get.
- Reserve "model" in the slash-skill reserved-name contract (backend,
contract fixture, frontend), register the Telegram handler, and
document the command in /help, README, and README_zh.
* fix(channels): fence model pin cache and mirror runtime model resolution
Review follow-up for /model:
- Serialize cold pin loads with persist+cache publication under a
per-manager lock, so a slow threads.get that captured pre-pin
metadata can never overwrite a concurrently committed selection
(commands are not run-serialized).
- _resolve_configured_model_name now replicates the run path's shallow
session merges and applies the real runtime precedence —
config.context (preserved by build_run_config, unbeatable by the
setdefault-filled body context) > top-level context > configurable —
instead of scanning per-layer carriers; a later layer replacing a
nested carrier wholesale no longer leaks a stale value into the
status reply.
- /api/models resolves the trusted bound owner for internal channel
calls and filters the list with the owner's role, so /model
selection gates on the same principal that run admission admits for
that account.
Tests: pin-write-vs-in-flight-load race, status compared against the
real merged runtime config (parametrized across carriers),
nested-carrier replacement, and owner-role filtering with the real
RBAC provider; each verified red by reverting exactly its guard.
* fix(channels): share model resolvers with the run path and harden the pin lifecycle
The /model status reply re-implemented the session-layer merge and the
Gateway carrier precedence, and it ignored the custom-agent model
fallback — on a thread pinned to a custom agent whose config sets
``model:`` it reported "server default" while the run actually used
the agent's model. Review also found the pin lock spanned a network
call and a stale pin bricked its conversation behind the generic
error fallback.
- Extract ``_select_carried_model_name`` as the single carrier-precedence
rule; the status reply reuses the run path's exact ``_merge_dicts``
layer shapes, so a future layering change has one place to touch.
- Extract ``_resolve_message_assistant_id`` / ``_resolve_assistant_id``
and share the assistant precedence chain between the run path and
the status reply.
- Fall back to the resolved custom agent's configured ``model`` when no
session layer carries one, mirroring the agent factory's
``_resolve_model_name(requested or agent_model_name)``; the reply now
distinguishes "(from channel configuration)" and "(from agent
configuration)".
- Run the cold ``threads.get`` in ``_load_thread_model`` outside
``_model_pin_lock``; only cache publication takes the lock, with a
post-GET re-check so an in-flight /model write still wins over the
older snapshot.
- Detect the run-admission allowlist rejection in ``_handle_message``
and recover: clear the stale pin (durable write plus cache under the
lock) and reply with the reset and the /model recovery instead of
the generic internal-error fallback.
- Document /model and the ``channel_model_name`` pin semantics in
backend/app/channels/AGENTS.md.
* fix(channels): fence stale-pin recovery and cover streaming admission failures
Review round 4 found the stale-pin recovery itself unsafe and
incomplete:
- Recovery cleared whatever pin was current when the delayed
admission rejection arrived, so a rejection for an old pin could
erase a replacement selection committed mid-flight (and tell the
user the new model was unavailable). ``_stale_pin_recovery`` now
takes the failed run's own selection (``expected_pin``, read from
its run context) and clears only while that selection is still
current under the lock; a replaced or already-reset selection gets
a non-destructive reply instead.
- Feishu/Telegram/DingTalk streaming turns never reached the recovery
hook because ``_handle_streaming_chat`` swallows start_run's 400
into its terminal-error reply. The recovery now runs in the
streaming terminal-error path too, delivered as the single final
outbound with the same dedupe cleanup; blocking, streaming, and
fire-and-forget run-creation paths all handle the rejection at the
site that knows the submitted selection.
* docs(channels): fit the /model entry within the agent-guidance size budget
The base file sits one byte under the AGENTS.md hard limit, so
documenting /model requires net-zero growth: keep the command-inventory
addition and a compact pin-semantics sentence, and losslessly tighten
the surrounding prose (the file was already at the limit).
* fix(channels): apply the review suggestions on /model replies
Three non-blocking findings from the latest review:
- The /models pin footer decorated the failure string too, advertising a
pin next to an error that lists nothing. The command now fetches the
listing through ``_fetch_gateway_json`` and appends the footer only on
a successful fetch.
- The /model status reply labeled every session-layer model "from
channel configuration", misreporting user- and default-layer models.
The label is now "session configuration" for every session layer.
- Stale-pin recovery replies read as if the turn went through, but the
run was rejected before starting and the triggering message never
reached the agent. All recovery variants now end with an explicit
"Your message was not processed — please send it again."
* fix(channels): recover stale model pins on the follow-up drain path
The drain is the fourth run-creation path and the only one without
stale-pin recovery: a /model pin that left the allowlist while follow-ups
were buffered made every drained runs.create fail admission, requeueing
the batch into the same 400 on every cycle with no user-facing recovery —
contradicting the documented "stale pins self-clear" invariant.
- Compute expected_pin from the drain's run context (mirroring
_handle_chat_on_thread, initialized None since _resolve_run_params is
itself inside the guarded try) and run _stale_pin_recovery on drain
failure before requeueing; the pin self-clears and the requeued batch
succeeds on the next cycle.
- Pass the caller-authoritative thread_id into _stale_pin_recovery:
every call site already holds it, so recovery never re-derives it from
the message — a store-mapping rewrite between dispatch and recovery
can never clear the wrong conversation's pin, and the lookup's
failure mode (and its guard) disappears entirely.
- Return StalePinRecovery(text, resolved) so requeueing callers can
tell a cleared pin from a repeat failure: the drain logs INFO when
the next cycle is free of this rejection and WARNING when recovery
could not remove it (persist failed, or the selection came from
session configuration, which still carries it) — no silent infinite
requeue loop. Drain logs carry structured facts only: the reply text
ends with a "please resend" note that would double-process buffered
messages retried automatically (GitHub, the only buffering channel,
sends log-only anyway).
- Reword the config-carried recovery reply: with no pin held, the
rejected selection came from session configuration — name that source
and the fix instead of claiming a default takeover that would not
happen.
- Correct the _load_thread_model lock docstring (writes deliberately
hold _model_pin_lock across threads.update) and read the agent config
for /model status off the event loop via asyncio.to_thread, with a
Blockbuster-gated blocking_io regression test.
- Pin the under-lock re-check in _stale_pin_recovery with a concurrency
test (a /model committed during the lock-wait window wins; recovery
persists nothing), mutation-verified — the drain is the only caller
running on a background task without the serialized-run lock, so it is
the path most exposed to that window.
* Revert "fix(channels): recover stale model pins on the follow-up drain path"
This reverts commit 0b4652c689afb56791e5e670901c97a4bf44a156. The drain
recovery work moves to a dedicated PR stacked on this branch, keeping #6534
scoped to the /model feature itself.
* fix(channels): recover stale model pins on the follow-up drain path
The drain is the fourth run-creation path and the only one without
stale-pin recovery: a /model pin that left the allowlist while follow-ups
were buffered made every drained runs.create fail admission, requeueing
the batch into the same 400 on every cycle with no user-facing recovery —
contradicting the documented "stale pins self-clear" invariant.
- Compute expected_pin from the drain's run context (mirroring
_handle_chat_on_thread, initialized None since _resolve_run_params is
itself inside the guarded try) and run _stale_pin_recovery on drain
failure before requeueing; the pin self-clears and the requeued batch
succeeds on the next cycle.
- Pass the caller-authoritative thread_id into _stale_pin_recovery:
every call site already holds it, so recovery never re-derives it from
the message — a store-mapping rewrite between dispatch and recovery
can never clear the wrong conversation's pin, and the lookup's
failure mode (and its guard) disappears entirely.
- Return StalePinRecovery(text, resolved) so requeueing callers can
tell a cleared pin from a repeat failure: the drain logs INFO when
the next cycle is free of this rejection and WARNING when recovery
could not remove it (persist failed, or the selection came from
session configuration, which still carries it) — no silent infinite
requeue loop. Drain logs carry structured facts only: the reply text
ends with a "please resend" note that would double-process buffered
messages retried automatically (GitHub, the only buffering channel,
sends log-only anyway).
- Reword the config-carried recovery reply: with no pin held, the
rejected selection came from session configuration — name that source
and the fix instead of claiming a default takeover that would not
happen.
- Correct the _load_thread_model lock docstring (writes deliberately
hold _model_pin_lock across threads.update) and read the agent config
for /model status off the event loop via asyncio.to_thread, with a
Blockbuster-gated blocking_io regression test.
- Pin the under-lock re-check in _stale_pin_recovery with a concurrency
test (a /model committed during the lock-wait window wins; recovery
persists nothing), mutation-verified — the drain is the only caller
running on a background task without the serialized-run lock, so it is
the path most exposed to that window.
(cherry picked from commit 0b4652c689afb56791e5e670901c97a4bf44a156)
* fix(channels): apply the third review round on /model command replies
- Delete the dead kind == "models" branch in _fetch_gateway: /models has
rendered the listing inline since the pin-footer fix, so the helper's
copy was unreachable and had already diverged (unguarded m["name"], no
pin footer). The internal-auth-header test now drives the surviving
/memory kind.
- Guard the inline /models listing comprehension like the
_handle_model_command sibling (isinstance(m, Mapping) and
isinstance(m.get("name"), str)), so a malformed payload entry can no
longer turn the command into the generic internal-error reply;
regression: test_models_command_skips_malformed_entries.
- Drop the unused user parameter from _resolve_bound_owner_for_model_list:
the resolved owner comes solely from the trusted header, and the
parameter implied the caller's own principal shaped the filter.
* fix(channels): apply the fourth review round on the /model lifecycle
- Preserve fresh model pins across cache eviction: a cold threads.get
racing a successful /model write could still publish its stale
snapshot after clear-on-pressure evicted the fresh entry — the
in-cache re-check cannot see an eviction. The model-pin cache now
carries a write generation; a load whose generation moved during the
GET refetches once instead of publishing over the committed
selection. Regression:
test_fresh_model_pin_survives_cache_eviction_during_cold_load
(blocked GET, write, eviction, then the refetch wins),
mutation-verified.
- Resolve context-carried agents before reporting their model: /model
status now mirrors the run path's effective-agent rule — explicit
message/thread choice overwrites every carrier, then the merged
carriers' agent_name (new _select_carried_agent_name, sharing the
model-name precedence via _select_carried_value), then the
session-layer assistant_id that only setdefaults into context. A
session with context.agent_name=coder now reports coder-model instead
of server default, and assistant_id=writer alongside
context.agent_name=coder reports coder-model instead of writer-model.
Regressions: context-carried, context-beats-assistant_id, and
thread-pin-beats-context status tests.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): confine review_skill_package local targets to the caller
Local path targets were allowed anywhere under the Gateway cwd or /tmp.
Every documented deployment keeps DEER_FLOW_HOME under the cwd, so a user
who knew another user's id could pass
.deer-flow/users/<id>/skills/custom/<skill> and get that skill's SKILL.md
and references/ back in the model response. The tool is always available,
so skill tool policy could not remove it.
Local targets are now confined to the configured skills root and the
caller's own user directory (resolved from the runtime user id through
make_safe_user_id, like the user-scoped skill storage). skill:// and
inline:// targets are unchanged.
* test(skills): pin dot-dot and sandbox-planted symlink review targets
* docs(changelog): reference #6580
* fix(skills): ignore a review skills root that contains per-user directories
A skills root configured at or above DEER_FLOW_HOME would re-admit every
user's directory as a local review target. Drop such a root and log a
warning instead of trusting it silently.
* test(skills): use the platform temp dir for the shared-temp review case
A hardcoded /tmp does not exist on Windows dev hosts.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs: list the summarize event tag and scope the extension example's claims
RUN_EVENT_STREAM.md listed seven middleware tags, but the catalog and the
run event stream contract carry eight; `summarize` is emitted by
SummarizationMiddleware for every successful summary result. Add the tag,
describe the event, and pin the documented list to MIDDLEWARE_EVENT_TAGS so
the next tag cannot drift silently. summarization.md now lists all eight
`changes` fields of that event.
The extension example said it showed all five (README, pyproject, module
docstrings, test name) or every contribution kind, while ExtensionRegistry
has eight. Say it demonstrates five, name the three it does not, point to
the observers guide and the bookmarks plugin, and assert in the example's
test that the two observer kinds stay unregistered.
* docs(changelog): reference #6582
* docs: clarify extension example package layout scope
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(events): preserve JSONL sequences after run deletion and restart
* fix(events): match JSONL watermark permissions to run files
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes
Why this change:
Two fail-open paths in the AIO sandbox lifecycle. First, the lark broker
capability probe collapsed "provisioner unreachable" into "broker not
configured": one 1.5s timeout cached a negative for 300s, so new sandboxes
were created without the broker sidecar while the per-user credential
directories were still bind-mounted into the container, and existing Pods
kept being reused without their actual mode ever being read. Second, an
implicit-shell command that timed out or returned an unknown status only
marked the current client object; warm-pool reclaim and rediscovery built a
fresh client that re-entered a container whose shell state could not be
proven clean, and nothing about that ambiguity survived a Gateway restart.
What changed:
- The capability cache stores confirmed observations only, keyed per
provisioner endpoint/auth identity; an unknown answer (timeout, malformed
payload) raises instead of selecting credential placement, while a
confirmed broker requirement is retained through outages. One in-flight
probe per identity shares success and failure, and a Pod-attested mode
contradicting the cache forces one shared refresh.
- Execution uses the mode attested for the actual sandbox:
SandboxInfo.lark_cli_broker flows from the provisioner's atomic Pod
observation into the client, so a probe outage delays drift detection but
never denies reuse of an attested runtime, and never trusts an unattested
one.
- The provisioner reports each Pod's actual broker mode, generation (Pod
UID) and shell capacity from one read, rejects lark requests conflicting
with its configuration (409/503 carrying a capability-refresh marker the
Gateway uses to drop the stale cache entry), rejects conflicting or
Terminating Pods on coordination creates while reporting Terminating as
data to status/get/list, and answers Pod presence independently of
Service readiness.
- A persistent quarantine (DEER_FLOW_HOME/sandbox-quarantine, keyed by
scope/sandbox/generation) fences containers after uncertain implicit
shell outcomes across client rebuilds and restarts; in-process reuse,
warm reclaim, discovery, create responses and orphan reconciliation all
pass one policy gate, and replacement runs under the existing ownership
and teardown fences. A fenced sandbox still tracked after a failed
release-recycle is destroyed inline on the next acquire.
- Quarantine records retire only after confirmed absence, retrying partial
cleanup for the quarantined generation and releasing remembered ports
under the same fences; local stop failures now propagate instead of being
logged away, and remote destroy errors surface to callers.
Problem solved:
A provisioner blip could silently strip broker isolation from new lark
sandboxes for minutes at a time, and ambiguous shell outcomes let corrupted
default shells persist across warm reuse, rediscovery and restarts. Docs
updated: README, aio_sandbox and skills AGENTS.md.
* docs(changelog): record sandbox attestation and quarantine fix (#6436)
* fix(sandbox): preserve active leases and fail closed on unattested broker modes
Concurrent uploads could recycle a healthy AIO runtime while another holder
was creating or cleaning up a command session. Older provisioner responses
could also trigger destructive replacement before failing admission.
- Distinguish pending session creation from ambiguous outcomes and reserve
idle teardown until execution and upload holders finish scope cleanup.
- Reject stale fork lookups and reconcile cancelled acquires without
releasing a sandbox still borrowed by another holder.
- Default Lark broker mode to unverified and preserve unattested Pods until
the provisioner is upgraded.
- Recognize Docker 29 absence diagnostics while matching resource names exactly.
- Preserve Gateway upload errors and cover admission and lifecycle regressions.
* fix(sandbox): scope the unverified-broker refuse path to attested providers
The fail-closed raise in _lark_cli_env_from_runtime treated every Sandbox
without a lark_cli_broker attribute as unverified, but only AioSandbox
carries the attestation — E2B, OpenSandbox, Tenki, and Boxlite have no
broker sidecar, no credential mounts, and no provisioner to upgrade, so
every lark-cli command on those providers failed with an unactionable
error.
Declare the tri-state on the Sandbox base class (defaulting to False,
the only mode brokerless providers have) so the None fail-closed path
stays scoped to AioSandbox's deliberate unverified default.
- Add lark_cli_broker: bool | None = False to the Sandbox ABC, mirroring
the persistent_shell_sessions tri-state pattern
- Add a regression test pinning the non-broker overlay for a provider
without the attribute
* fix(sandbox): close review gaps in broker attestation and teardown lifecycle
Address the latest #6436 review round:
- The provisioner's post-build broker-mode 409 now carries the
capability-refresh marker, so an incompatible-mode conflict surfaced by
idempotent re-create, attestation drift or post-create verification drops
the Gateway's cached capability observation instead of repeating the same
409 for the full negative-cache TTL.
- AioSandboxProvider.destroy() returns the _destroy_tracked outcome: False
when teardown is deferred because an execution/upload holder or a peer
teardown is still active, so callers can no longer mistake a deferred
destroy for a completed one.
- A live replacement under the same logical ID proves the fenced old
generations are absent; retire_replaced_generations prunes only those
stale records so they cannot strand future creates, while the live
generation's fence and unversioned crash fences are never touched.
- SandboxInfo.from_dict keeps a missing lark_cli_broker attestation
unverified (None) instead of silently rehydrating as attested non-broker.
- The restricted-network inspect path shares the strict
_is_no_such_network_error parser, closing the substring loophole for
context-name collisions and wrong-target errors.
* docs(sandbox): condense lease notes to keep the AGENTS.md chain under the size gate
The three lease notes added in 26c307f3 pushed the effective guidance chain
to 98,312 bytes — 8 over the 98,304 hard limit enforced by agent-guidance
(AG002). Fold them into two lines carrying the same rules.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): number remote read_file line ranges like LocalSandbox
E2B, BoxLite, Tenki and OpenSandbox sliced ranged reads with str.splitlines(), which also ends lines at a bare "\r", \f, \v, \x1c-\x1e, \x85, \u2028 and \u2029. LocalSandbox.read_file, count_file_lines and the read_file truncation marker end lines only at "\n", so on those providers a ranged read of such a file returned different lines, and following a truncation marker's start_line went back into text already shown. Add split_file_lines next to count_file_lines and use it in the four providers.
* test(sandbox): separate remote and host read paths
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Opt-in per-document summary column rendered into the injected
<documents> index, list_project_documents JSON, and the documents API.
Pipeline (projects/summaries.py, behind the narrow enqueue_summary
interface):
- The two shelf write routes enqueue a job on created=True; dedup hits
never regenerate. Off by default (projects.summaries_enabled).
- Bounded worker pool (summary_concurrency + bounded summary_queue_size;
a full queue drops the job) fed by FIFO; shutdown drains queued jobs,
delivers one sentinel per worker via blocking put (queue smaller than
the worker pool cannot strand idle workers), and bounds only
genuinely in-flight jobs with the grace period.
- Jobs carry (document_id, project_id, user_id) explicitly; the job's
owner is bound into the user ContextVar around the model call so
tracing/usage attribution never falls back to 'default'.
- Text source reuses read_text_serving_path (convertible-first rule,
content validation); conversion has no deadline (await_drained), the
model call has a caller-side wait limit — the worker slot is retained
until the provider call actually finishes and late results are
discarded.
- Output contract: strip leading <think> reasoning -> unwrap fence/
quotes -> collapse to one line -> empty is NULL -> UTF-8 byte cap on
a character boundary; length-capped responses (provider-normalized
detectors) are failures.
- set_summary pins updated_at (literal self-assignment) so a landing
summary never reorders the shelf; owner-predicated UPDATE is a no-op
on rows purged mid-generation.
- Metrics: eligible/success/failed/skipped/queue_dropped/
uncapped_accepted.
Schema/config:
- project_documents.summary (migration 0036, nullable, no backfill;
forward-compat floor unchanged).
- Six projects.summary_* keys, all registered in STARTUP_ONLY_FIELDS
(captured once at lifespan startup); config_version 57 -> 58.
- run_oneshot_llm gains an optional model_overrides passthrough (passed
only when set, existing callers unchanged) and a detailed variant
exposing the raw response message for termination-signal detection.
Docs: API.md response field, ARCHITECTURE.md pipeline paragraph,
migrations AGENTS.md chain head, root AGENTS.md records the scoped
all-core exception with the lift-out path.
* feat(goal): record the met goal and keep goal state on the history head
- Add a goal_outcome LastValue channel (GoalOutcomeState). Only the
guarded clear of a satisfied ordinary goal writes it, in the same
checkpoint as the clear; the reply anchor is the latest visible AI
message with text, the reply the visible signature keys on. A goal
stored without created_at still clears and gets a record.
- write_thread_goal removes the record on every other goal write and
lists it in channel_versions, new_versions and metadata writes when
it changes. Threads that never had a record keep their checkpoint
shape. The sync-only SqliteSaver and CachedHistorySaver, which take
the #6448 sync fallback, store and drop it the same way.
- goal_outcome is server-owned: POST /state and non-internal run input
that carry it get 400. A POST /state goal write, or a goal in
external run input, clears it.
- A branch head written onto its replay base resets the channels the
head lacks, so a goal or record that write_thread_goal removed does
not come back. Branching from the met turn keeps the record without
the met goal.
- The POST /history head now carries an active goal and an achieved
goal_outcome, also before any message exists, so the post-run
history refetch keeps them. Older entries stay title/thread_data.
- contracts/thread_goal_contract.json pins the chat stand-down and
check-failure codes, the record shape and status, the head keys and
the host-written reasons.
* feat(frontend): show why a goal stopped and when it was met
- The goal bar derives its state from the active goal, the goal_outcome
record and the messages: set, in progress, continuing, stopped (limit,
auto-continue off, no progress, token budget, missing evidence, failed
run, unknown code), waiting for you, waiting, couldn't check the goal,
paused and goal met. An 8/8 stand-down now reads "Stopped" with the
next step instead of "Continuing 8/8", and a count-0 stand-down no
longer looks like a fresh goal.
- "Goal met" shows while its anchor reply is the latest visible message
(a visible human message, or a visible AI message with text, moves the
chat on), so a reload agrees with the live view. An active goal wins
over the record, and this tab's /goal set or clear hides it.
- useActiveGoal drops the optimistic goal when the record of that same
goal arrives, which fixes the bar that stayed after a goal met on the
first run (the clear frame has no goal key), also for a goal stored
without created_at. Server goal values pass through isGoalState; a
/goal status read uses the backend's own active check, so the bar, the
toast and the edit lock agree with the server.
- Row 2 gives the reason and what to do next; Details holds the full
goal, the checker's own note (never host-written text, a check failure
or a paused goal's older check), the assumption note and an unknown
code, and starts collapsed in each new state. The bar is a labelled
section with a live region that announces state changes only (another
goal's same state counts) and re-reads its text after a language switch.
- The edit pencil on the latest turn is shown disabled with "Editing is
off while a goal is set" instead of hidden; the message toolbar shows
on focus-within, so keyboard users can reach it.
- A /goal set or clear rejected with 409 keeps the draft and says a run
is still going; other failures keep the server's message and the draft.
A /goal request replaced by a newer one leaves the composer alone, so
a double submit no longer empties it before the second one fails.
- Copy lives in inputBox.goalBar (en-US, zh-CN); a contract test reads
contracts/thread_goal_contract.json, and the raw-identifier guard now
catches stand-down codes and goal field names.
* test(frontend): cover the goal bar on both chat pages
- tests/e2e/goal-status.spec.ts runs every case on the chat page and the
agent chat page, since each wires the goal bar and the edit lock itself:
a goal stopped at 8/8, one waiting for details and one waiting on an
open question card show their reason and next step, also after a
reload; the locked edit pencil gets keyboard focus, shows its toolbar
and tooltip, does not open the editor, and hides while a file uploads;
Goal met after one automatic follow-up (In progress, Continuing 1/8,
then "Auto-continued once") survives a reload and hides after the next
visible message; a goal met on the first run of a new chat replaces the
set bar and frees the pencil, and a new /goal replaces Goal met while
its own run goes on; a /goal set or clear refused with 409 keeps the
draft for the retry and shows its own busy toast.
- The mock POST /history head carries only the contract's
history_head_keys, with goal only while active and goal_outcome only
when achieved; thread state and run values omit an unset goal instead
of sending goal: null. PUT and DELETE /goal drop the met record,
setGoalWritesBusy returns the Gateway's 409, and a repeat create of a
requested thread id returns the thread as is, like the Gateway.
- The raw-identifier guard also catches checker blockers and the met
record's field names.
- frontend/src/AGENTS.md: the GoalStatus sentence now names goal,
goal_outcome and useActiveGoal, reworded in place.
* test(frontend): keep the draft through a failed upload and a busy /goal
#6412 fires onSent only once a send is dispatched, so a failed upload
keeps the text, files and stored draft. A /goal the Gateway refuses with
409 never reaches sendMessage, and the composer rejects it so PromptInput
keeps the text. Drive both through the real sendMessage in one composer:
a failed upload keeps "summarize it" stored, a busy /goal keeps its text
and stored draft with no run, and the retry saves the goal and starts
one run with the objective, not the "/goal" text. After that run the
composer is empty and no stored draft comes back.
* fix(frontend): drop the stop condition's closing period before the pause clause
A stored stop condition often ends with a period, so the schedule card
read "…remain)., pauses itself" (and "…。，满足后自动暂停" in Chinese).
Trim the condition's closing sentence punctuation before the clause that
follows it. The event line, where the condition ends the sentence, is
unchanged.
* fix(frontend): point "evidence missing" at the agent, not a user reply
"Evidence missing" asked the user to reply with what's left, which is
the push a goal is meant to save. It now asks the agent to show the
result or explain what's missing, and offers /goal clear only for a goal that
no longer applies: clearing drops the met record, so work that is right
would never show "Goal met".
zh-CN follows. Copy only: no key, state or mapping changes, and the
scheduled page's reason labels are untouched.
tests/unit/components/workspace/goal-status-next-steps.test.ts checks
the row in en-US and zh-CN, and that no stop row offers /goal clear
without that condition. goal-status.dom.test.tsx and
goal-stream.dom.test.tsx now expect the new copy.
* fix(frontend): say "Couldn't check the goal" once in the goal bar
After a goal check that could not run, the bar's second line borrowed
the scheduled page's whole reason, so it repeated the chip: "Couldn't
check the goal · Couldn't check the goal; the run itself may be fine.
It's checked again after your next message." Chinese had the same
repeat.
The line now gives only why the check failed ("The run itself may be
fine", "No final reply was saved" or "The chat changed during the
check"), then when it is checked again. The screen-reader announcement
follows. The chip and the scheduled page's reason labels are unchanged.
tests/unit/components/workspace/goal-status-unchecked.test.ts checks
every check-failure code in en-US and zh-CN: the label appears once,
and the line still says why the check failed.
* feat(goal): redact the evaluator input when pii_redaction is enabled
The goal evaluator calls its model directly, outside
PiiRedactionMiddleware, so it sent raw user lines, tool arguments, tool
results and Human Input Card answers even with pii_redaction enabled.
With pii_redaction enabled:
- the messages in the evidence window are redacted whole before the
evidence caps can cut an identifier in two, as TitleMiddleware does.
This reuses redact_queued_messages, which already covers content,
call arguments and card answers. Earlier messages are left alone: the
evaluator never sees them, and the pass is synchronous work on the
event loop (0.84 s for an 800-message, 4 MB thread, 0.06 s for its
window);
- the assembled input, the goal objective included, is redacted again.
A redaction error fails the check with evaluator_failed; raw text is
never sent instead. When redaction is off, the input is byte-identical.
Thread state keeps the raw text. README.md, config.example.yaml and the
middleware docstring list the evaluator with the other redacted inputs.
tests/test_goal_pii.py puts an address where the evidence caps cut a
tool argument, a tool result and a card answer, and checks the
objective, the unchanged input when redaction is off, that only the
evidence window reaches the redactor and gives the same evidence as
redacting the whole thread, that a redaction error stands down with
evaluator_failed before the model is called, and the worker path.
* fix(goal): redact the hidden continuation when pii_redaction is enabled
The hidden /goal continuation repeats the goal objective and the
evaluator's reason and evidence summary. PiiRedactionMiddleware skips
it, since it is a framework message and not a user message, so with
pii_redaction enabled each continuation turn sent them raw to the
agent's model, and the message stayed in the thread for later model
calls. The durable-context block and the evaluator input already redact
the objective.
With pii_redaction enabled, make_goal_continuation_message now redacts
the objective, the reason and the evidence summary with redact_text
before the message is built. The thread keeps the redacted message,
which the UI hides; the goal state keeps the raw objective.
The worker builds the message before it counts the continuation. A
redaction error fails the check the way an evaluator redaction error
does: run_failed with stand_down_reason evaluator_failed, the goal stays
active, nothing is sent and the continuation budget is not spent. Only
the exception type is stored. With redaction off, the message is
byte-identical.
README.md, config.example.yaml and the middleware docstring list the
continuation with the other redacted inputs.
tests/test_goal_continuation_pii.py checks the redacted objective,
reason and evidence summary against the tokens redact_text gives, the
unchanged message with redaction off or no config, the worker path
under the run config, and a redaction error.
* docs(changelog): link the goal outcome entries to #6556
Fill the four ([#PR]) placeholders in CHANGELOG.md and CHANGELOG_zh.md with this PR's number and add the matching link references.
* fix(frontend): keep an active goal stored without timestamps visible
POST /threads/{id}/state and run input can store an active goal without
created_at or updated_at. The backend still evaluates it and rejects an
edit with 409 (is_active_goal checks only status "active"), but
isGoalState required both timestamps, so history and live values
dropped it: both chat pages hid the goal bar and unlocked the last-turn
pencil, and only a /goal status read, which used a looser predicate,
brought them back.
isGoalState now accepts any record with a string objective and status
"active", and normalizeGoalStatusRead uses it, so thread values, live
updates and status reads share one rule. Malformed or inactive values
are still dropped.
Tests cover a sparse active goal from the history head and from a live
update before the post-run refetch (bar shown, hasGoal true), and a
malformed goal that stays hidden.
* test(goal): check the outcome write metadata the same way on every saver
PostgresSaver stores checkpoint metadata through
get_serializable_checkpoint_metadata, which drops "writes", so the
[postgres] case of test_goal_outcome_round_trips_through_the_sync_fallback
failed with KeyError: 'writes' when CI ran it against a live database.
The test now records the metadata write_thread_goal passes to put and
asserts the clear's and the next goal's "writes" there, for every saver.
The stored channel values are checked as before, and on sqlite and
cached-sqlite the stored "writes" must still equal what was passed.
* docs(agents): keep goal_outcome out of the agents guidance chain
The middlewares AGENTS.md chain is already over the 98,304-byte AG002
hard limit on main, and the PR-mode check fails any chain a PR touches,
so adding goal_outcome to the ThreadState field list in
deerflow/agents/AGENTS.md failed the agent-guidance job. goal_outcome
stays documented in runtime/AGENTS.md and backend/app/gateway/AGENTS.md,
which are outside that chain.
* refactor(goal): read the evaluator's evidence and redaction from one window
format_visible_conversation recomputed the window start that the redaction
pass takes from _evidence_window. Both now use _evidence_window, so a later
change to the cap or to what counts as visible moves both.
The window test now fills the window with tool-only replies and tool results.
Before, it caught a cap change in one copy but not a change to what counts
as visible.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(community): give max_results coercion a single owner
#5865 audited every bundled provider's max_results coercion, and #5866 /
#5867 are fixing seven copies of it. Each one had to be read on its own
because four of the copies were byte-identical apart from the provider
label in their warning line.
Move the shared bar into deerflow.community.search_max_results and have
ddg_search, image_search, fastcrw and firecrawl delegate to it. Warnings
keep their existing text and their existing logger: logger is a parameter
so records keep their module attribution, which
tests/test_ddg_search_tools.py filters on.
image_search also stops spelling its own default twice -- the two tool
signatures hardcoded 5 while the same file defined DEFAULT_MAX_RESULTS.
No bound is added to any provider, and brave / groundroute / serper /
serply / sofya / tencent_wsa are left untouched so #5866 and #5867 land
without conflict. The new test pins the 13-value behaviour, the delegation
and the historical labels, and asserts that the set of providers still
hand-rolling a coercer is exactly the declared six -- so a twelfth copy
fails CI rather than being audited by hand later.
* test(search): cover generic and inline max-results coercion
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The existing cross-thread test only goes red on Python 3.13+, while every
backend CI workflow pins 3.12, so removing the yield added in #6565 would
stay green. Record a middleware event from a raw thread, join it with no
intervening await so the call_soon_threadsafe hop is provably pending, then
close(flush=True): without the yield the event is dropped on every version.
Follows up the review on #6565.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(runtime): keep thread checkpoints when a rollback cancel precedes snapshot capture
A rollback cancel that lands after try_start() flips the run to running but
before _capture_rollback_point() completes left rollback_point=None with
snapshot_capture_failed=False. The rollback finalizer treated that state as
"the thread never had a pre-run checkpoint" and called adelete_thread(thread_id),
deleting the whole conversation's checkpoints.
Initialize snapshot_capture_failed fail-closed and clear it only after the
capture completes. A completed capture that returns None still resets an empty
thread as before. Covered by two regressions in tests/test_run_worker_rollback.py
(cancellation before and during the capture await).
* fix(runtime): drop the redundant snapshot_capture_failed assignment
The fail-closed initializer owns the capture state: `snapshot_capture_failed`
is True from the top of run_agent, and the capture block runs once, so the
except branch can only observe it still True. Re-assigning it there was dead
code (review nit from @willem-bd).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): classify duration-only checkpoints without the Postgres-dropped marker
On Postgres the savers funnel metadata through langgraph's
get_serializable_checkpoint_metadata, which pops writes before the row
lands, so persist_run_history_metadata's runtime_run_duration marker never
comes back and every replay/history read path treats metadata-only duration
leaves as addressable conversation state.
is_duration_only_checkpoint now classifies in two tiers:
- writes present (memory/SQLite) is authoritative — the per-writer key is
exact, so a title leaf carrying runtime_interrupt_title with inherited
run_durations is not a duration checkpoint;
- writes absent (Postgres) falls back to the surviving stamps
(source == run_durations / run_message_ids keys).
Stamps alone are not exact: _ensure_interrupted_title and
_rollback_to_pre_run_checkpoint copy the head's metadata and inherit them.
Read paths tolerate that because every stamp-bearing leaf either copies its
parent's messages verbatim or restores an ancestor's exact message state, so
a skipped leaf only advances a message-presence scan onto a state-equivalent
checkpoint. Destructive callers opt out via stamp_fallback=False:
enforce_thread_retention keeps widening the class only through the
parent-shape check in _mark_duration_leaves_without_the_marker.
* fix(gateway): confirm the duration stamp fallback by parent shape
Review on #5988 surfaced a real false positive: write_thread_goal copies
the head's metadata and thereby inherits the run_durations/run_message_ids
stamps while changing the goal channel, and on Postgres its own writes.goal
marker is popped too — so the stamp fallback classified the goal leaf as
duration-only and the branch scan walked past it, copying channel values
from before the goal write.
The stamp fallback now also requires the duration writer's shape: it clones
the parent's channel_versions verbatim, while every state-changing leaf
(title, goal) bumps at least one version. Callers resolve the parent from
their own context — the lineage walk fetches the grandparent only for
stamp candidates, the history-window scans (branch, latest-turn,
regenerate filter, chronological fallback) index the window they already
hold — and stamps without a resolvable parent are refused: scanning one
state-equivalent copy is free, skipping a goal leaf is not.
Also corrects the mechanism claims from the first pass: langgraph's
update_state paths (rollback, compaction, manual updates) persist metadata
built from scratch and cannot inherit the stamps at all; the inheriting
writers are title and goal.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(gateway): share the stamp gate and reuse the walked grandparent
Review round 2 on #5988 — all three items, no functional changes:
- extract _has_postgres_stamps as the single gate shared by the classifier
fallback tier and the walk probe, so widening one widens the other;
- carry the grandparent fetched for a stamp candidate into the next
iteration (it is exactly the next parent), turning a chain of N Postgres
duration leaves from ~2N+1 ancestor reads into N+1;
- pin the unresolvable-grandparent promise at walk level: a stamp
candidate whose own parent link was pruned stays addressable and remains
the replay base.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(gateway): resolve persisted channel versions for the duration stamp fallback
Materialized accessor snapshots expose no checkpoint payload, so the verbatim-copy shape check always failed open and Postgres duration leaves stayed addressable. Carry the version maps alongside the snapshots (raw snapshots expose them directly, graph-backed ones re-read the raw tuple through the accessor) and use them at every replay-base scan site.
* perf(gateway): resolve stamp-fallback versions lazily and once per tuple
The Postgres stamp fallback needs the raw channel versions for its shape check, and both window scans resolved them eagerly for every entry: a branch request paid up to 2N serial roundtrips for a window of N, and the regenerate scan resolved the same tuples a third time through resolve_history_versions.
Resolve them only for stamp candidates - the marker tier and the stamp gate decide without versions - and memoize by config identity, so a marker-decided window pays nothing and each distinct tuple pays at most one raw read.
The lineage walk had the same shape: a stamp candidate the shape check refuses re-read its grandparent as the next iteration's parent, because the prefetched slot was only filled when the candidate was skipped. Ancestor reads are now memoized by config identity, so no ancestor is fetched twice.
Dropped the dead checkpointer parameter of find_checkpoint_before_message_chronologically: no caller passed it, and wiring one in from an async handler would put a blocking get_tuple on the event loop.
---------
Co-authored-by: zeng-bohan <310902929+zeng-bohan@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* fix(uploads): skip files that vanish mid-scan in list_files_in_dir
The listing iterates scandir entries and stats each survivor. A
concurrent upload delete or the staging sweep can remove a file in
between, and the resulting FileNotFoundError escapes as a 500 on the
listing endpoint. is_file() does not cover this: on POSIX it answers
from the scandir d_type without touching the filesystem again.
Catch OSError around the stat and skip the entry on ENOENT/ENOTDIR/
ELOOP, the same tolerance the chmod path just above already applies to
replacement races. Operational errors such as EACCES still propagate.
Regression tests pin both directions: a vanished entry is skipped while
the rest of the listing survives, and a permission error still raises.
Fixes#6097
* fix(uploads): log vanished entries and fix comment per review
Address willem-bd's review on #6533: the skip now emits a debug line so
"my file disappeared from the list" reports are diagnosable without
changing the response contract, and the race comment names the actual
concurrent removers (the DELETE endpoint, possibly from another
replica, and sandbox processes) instead of the staging sweep, which
cannot remove a listed non-staging file.
The directory-level race between the is_dir() guard and os.scandir()
stays out of scope; it is covered by the still-open #6096 and its PR.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(uploads): tolerate directory races during listing
* docs(uploads): scope listing guidance to avoid inherited size limits
* fix(uploads): log directory races during listing
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Since CPython gh-105836 (backported to 3.13) an awaited executor future can
complete without a loop iteration when the worker finished before the future
was chained, so a middleware event a worker handed to the owner loop with
call_soon_threadsafe may still be pending when flush() starts. The run-end
flush() -> _detach_runtime_dependencies() sequence then dropped it. Yield to
the loop once before draining and re-check _closed.
Fixes#6564
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(helm): let the sandbox provisioner run with several replicas
The chart hard-coded replicas: 1 into the provisioner Deployment, gave it
no rollout strategy and no PodDisruptionBudget, and values.yaml had no
provisioner.replicas key, so a multi-replica gateway deployment still sent
every sandbox create/discover/destroy call through one Pod: while that Pod
restarted or its node drained no sandbox could be created anywhere.
- values.yaml provisioner gains replicas (default 1, unchanged), the
gateway's surge-then-drain RollingUpdate strategy and
podDisruptionBudget {enabled: true, minAvailable: 1}.
- provisioner-deployment.yaml renders replicas and strategy from values; at
1 replica the result equals the Kubernetes default strategy.
- New provisioner-pdb.yaml, rendered only while the provisioner is enabled,
the budget is enabled and replicas > 1, selecting on the Deployment's
labels and named <fullname>-provisioner so it never collides with the
gateway budget. No multiInstance override: nothing else in the chart
renders differently per provisioner replica count.
- docker/provisioner/app.py is unchanged. It keeps no state between
requests: the labelled sandbox Pods and Services are the only registry,
create tolerates the 409 a concurrent creator produces, destroy tolerates
404, NodePorts come from the API server and there is no background loop.
The chart README and docker/provisioner/README.md record this, and
tests/test_helm_provisioner_replicas.py pins it at source level next to
the values, template and helm-render assertions.
* docs(changelog): link the provisioner replicas entry to #6543
* fix(helm): preserve percentage PodDisruptionBudget values instead of casting them to 0
Review of #6543: both PDB templates rendered minAvailable through
`| default 1 | int`, and sprig's int turns a percentage string such as
"50%" -- a value Kubernetes accepts -- into 0, a budget that permits every
Pod to be evicted. The gateway budget had the same pre-existing cast.
- New shared helper deer-flow.pdbMinAvailable in _helpers.tpl: a YAML or
--set number renders as an integer, a digit string from --set-string
becomes that integer, a percentage is preserved quoted, and 0, "0%",
more than 100%, a non-integer or anything else fails the render with a
message naming the values key and the enabled switch to use instead.
Unset still falls back to 1.
- gateway-pdb.yaml and provisioner-pdb.yaml render minAvailable through it.
- values.yaml comments, the chart README multi-replica note and the
CHANGELOG entry say integer-or-percentage and that zero is rejected.
- tests/test_helm_provisioner_replicas.py pins the helper and the two
templates at text level and adds render cases for both budgets: "50%"
with three replicas renders "50%", 2 via --set and --set-string renders
2, and abc / 0 / 0% / 150% fail naming the key.
* fix(helm): normalize json.Number before the minAvailable string branches
Review of #6543: Helm 3.18.0 loads YAML numbers as json.Number, whose
kind is "string" but which regexMatch and the other string-typed sprig
functions reject at argument validation ("expected string; got
json.Number"). deer-flow.pdbMinAvailable gated its string branch on
kindIs "string", so the unoverridden chart default minAvailable: 1 made
every gateway.replicas=2 / provisioner.replicas=2 render fail on that
release, inside the README's documented 3.8+ support window.
- The helper keeps the explicit float64/int64/int branch and now converts
everything else with toString into $text before the digit, percentage
and fail branches; regexMatch, atoi, trimSuffix and quote only ever see
that plain string. A json.Number 1 renders 1, a json.Number 1.5 still
fails (the digit regex rejects it) and the hint still prints the raw
value; bools and other kinds still fail. The helper comment records the
Helm 3.18.0 reason.
- tests/test_helm_provisioner_replicas.py pins that the string branch
runs on the toString-normalized variable, that no string-typed function
is applied to $raw and that kindIs "string" is gone, and adds an
explicit render case for both budgets with only the replica count set
(the unoverridden default must render minAvailable: 1), marked as the
Helm 3.18.0 json.Number regression guard.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): send the CSRF pair on IM /goal writes
`/goal <objective>` and `/goal clear` from an IM channel sent their
PUT/DELETE to the Gateway with only the internal auth headers. The
Gateway's CSRF middleware does not exempt internal auth, so with auth
enabled (the default) both writes were rejected with 403 and the channel
replied "Failed to set goal." / "Failed to clear goal.".
`_goal_request` now sends the same double-submit cookie/header pair as
the channel's LangGraph SDK client; both build it from one helper,
`ChannelManager._csrf_headers()`.
* docs(changelog): reference #6537
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
When Sandbox.connect() fails with a trusted 'sandbox not found' error,
_reconnect_live_client() used to release the deployment-level slot but
still re-raise. _evict_oldest_warm() then treated that confirmed-gone
result like an unknown-state failure: it kept the eviction tombstone and
never called _end_transition_locked(), so the local transitioning slot
leaked on every retry against an already-reaped VM. Once the leaked slots
reach the configured replicas, every acquire rejects or times out with
SANDBOX_CAPACITY_EXCEEDED while active=warm=reserved=0.
Return None from _reconnect_live_client() for trusted gone errors so all
callers (eviction, warm reclaim, discovery, reconciliation) take their
existing already-gone cleanup paths, keeping local bookkeeping consistent
with the deployment-level release on the same signal. Unknown-state
errors (network, auth) still retain the slot for a later retry.
Also report the transitioning count in SandboxCapacityExceededError
details and the wait-timeout message so hidden slot usage is visible
without a restart.
Fixes#6540
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): avoid executor starvation in file-gate waits
Keep async file-gate waiters out of the executor needed by the holder.
Preserve sync and cross-loop exclusion, and drain started file work on cancellation.
* test(agents): cover mixed file-gate callers
Document why release wakes every async waiter.
Cover a synchronous tool holder and asynchronous waiter through the middleware entry points.
* test(agents): cover sync waiters released by async file tools
Signed-off-by: LingYi <lingyi01525@gmail.com>
---------
Signed-off-by: LingYi <lingyi01525@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`_local_launcher_home()` was the last caller of `shutil.which("bash")` left
behind by #5404, so on Windows it started the WSL launcher instead of Git Bash
and all 30 tests built on that helper failed while decoding the stub's UTF-16
output. Route it through `require_script_bash()` like the other shell-script
tests.
Git Bash also answers with MSYS paths (Windows TEMP is mounted at `/tmp`), so
the home printed by the launcher could not be used from Python. Convert with
`cygpath -w` inside the shell, where values that look like shell syntax stay
intact, and keep `printf` for hosts that ship no `cygpath`.
The unquoted `~/...` case additionally needs `USERPROFILE`: Git Bash expands
`~` from `HOME`, while Python's `expanduser()` reads `USERPROFILE` on Windows.
Windows: `tests/test_support_bundle.py` goes from 30 failed / 59 passed to
89 passed. POSIX hosts keep their previous behavior.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): decode Git Bash output as UTF-8 so CJK tools are not garbled
Git Bash/MSYS always writes UTF-8 to its pipes, but the MSYS branch of
LocalSandbox.execute_command called _run_windows_command without an
encoding, so the capture fell back to locale.getpreferredencoding(). On a
non-UTF-8 host (GBK on zh-CN Windows) that decodes UTF-8 bytes with the host
code page and mangles every non-ASCII tool result. The pipe decoder replaces
instead of raising, so the corruption is silent. The PowerShell branch right
next to it has pinned UTF-8 since #5440.
Decode the MSYS branch as UTF-8 too, and pin the behaviour in
tests/test_local_sandbox_encoding.py: a unit test for the decoder choice and
a live round-trip through a real Git Bash (skipped when it is absent).
* fix(sandbox): keep locale fallback for native-child output under Git Bash
Git Bash/MSYS writes its own output as UTF-8, but Windows-native children
(python.exe, CLI shims) inherit the pipe and keep writing the host code
page. Forcing UTF-8 with errors=replace turned that output into U+FFFD.
Decode the captured buffer as UTF-8 first and retry with the locale when it
is not valid UTF-8, so both MSYS UTF-8 output and native-child code-page
output survive. The PowerShell branch is unchanged (its own output is pinned
to UTF-8 by the preamble); the fallback is only wired on the MSYS path.
Adds native-Python CJK regressions through execute_command.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(auth): key the compose login lockout by client, not by nginx
Failed logins are counted per client IP, and the Gateway honors X-Real-IP
only from a peer listed in AUTH_TRUSTED_PROXIES. The compose files never
set it, and every browser request reaches the Gateway from the nginx
container, so all logins shared nginx's address: five wrong passwords
from anyone locked every user out for five minutes.
AUTH_TRUSTED_PROXIES now also accepts hostnames, resolved off the event
loop and cached for 10 seconds (failed lookups included, and only when
X-Real-IP is present), and both compose files default it to the bundled
nginx service, whose address Docker assigns. nginx overwrites X-Real-IP
with $remote_addr on every Gateway route; a test pins that for the three
maintained configs.
* docs(changelog): reference #6519
* fix(auth): share one trusted-proxy lookup across concurrent cache misses
A burst of setup-status / login requests on an expired hostname entry
started one getaddrinfo per request while DNS was slow, occupying the
default executor that password verification also uses. The cache check
and an in-flight task registration now happen without a yield between
them; later callers join that task through asyncio.shield, so a cancelled
request cannot cancel the lookup the others joined, and the task caches
its own result.
The compose default keeps `:-` on purpose: every browser request arrives
through nginx, so an empty AUTH_TRUSTED_PROXIES would only bring back the
shared lockout. The compose comment and .env.example say so, and the
rendered-compose test now covers the explicitly empty value.
* fix(docker): keep a .env AUTH_TRUSTED_PROXIES under make docker-start
docker.sh start runs Compose from docker/ without --env-file, so the dev
compose file's ${AUTH_TRUSTED_PROXIES:-nginx} interpolated from the shell
only, and the resolved environment entry outranked the operator's value
loaded through env_file. Export it from .env alongside the proxy variables
(a shell export still wins), and strip a CRLF before the quotes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The two Windows-only cases in tests/test_mcp_file_migration.py asserted a
literal "file:///C:/" / "file://C:" prefix on a path built from the pytest
temp dir. On a host whose checkout or temp dir lives on another drive both
assertions fail, so the only coverage of the drive-qualified file-URI forms
handled by _local_uri_to_virtual_path and _rewrite_local_paths_in_text never
ran there.
Derive the expected drive from the fixture path instead. The URI forms under
test are unchanged, so coverage holds on any drive.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): selectively reconcile changed deployment servers
* fix(mcp): validate the frozen task snapshot before installing epochs
Durable background calls resolve their connection from the configuration frozen
at Gateway startup, so installing a newer revision's binding epoch fenced every
status/cancel call with `StaleMCPBindingError` — even though hot reload of that
server is correctly rejected. The rejection only landed in `get_mcp_tools()`,
which runs after the cache had already installed the epoch.
Both decisions that install an epoch now consult the frozen snapshot first. The
reconciliation planner falls back to the conservative full reset, where callers
that still run against the startup configuration rebind in the replacement pool,
and never installs the rejected epoch. A discovery claim raises the same
`McpTaskConfigurationError` that `get_mcp_tools()` would, before touching the
pool and without recording the revision as applied. Unrelated non-task servers
still reconcile selectively.
Adds the three requested regressions — a rejected hot change keeps background
task callers working, a rejected revision installs neither epoch nor tombstone
(changed connection and disabled server), and an unrelated server change stays
selective — plus mutation proofs for both new gates.
* fix(mcp): always tear down a pool the claim path retired
`reset_session_pool()` only fences and unlinks the singleton; the owners are
signalled by `close_all_sync()`. The claim ran that teardown *after* its critical
section, so a claim that retired the pool for a new shared-reset generation and
then raised — a revision the frozen durable-task snapshot rejects is the
reachable case — left the retired pool's sessions running without ever receiving
a close signal.
Wrap the claim in `try`/`finally` so the retired pool is torn down on every path,
still outside every lock and after the lock is released. Adds a regression
covering the shared-reset-then-rejected-revision sequence: the error propagates,
the retired pool's owner observes exactly one `__aexit__`, the replacement pool
holds no rejected epoch, and restoring the startup configuration initializes
normally again.
* fix(mcp): deliver retired-pool cleanup before the cache reader waits
A synchronous `get_cached_mcp_tools()` reader can apply a full reconciliation
while a discovery is in flight — an interceptor change and a new shared-reset
generation both qualify, because the applied baseline exists before any cache is
published. `reset_session_pool()` only fences and unlinks the pool, and the
reader then entered its wait/retry branch from inside the condition, skipping the
`close_all_sync()` that signals the retired pool's owners. The replacement cache
published normally while the discarded pool kept a live owner and stdio
subprocess that no later reset could reach.
Hoist the observed waiting generation out of the critical section and deliver the
retirement before every wait/retry path, then re-acquire the condition and wait
on that exact generation so the predicate cannot miss a wakeup. Cleanup stays
outside `_init_condition`, and the early `return _mcp_tools_cache` cannot skip a
retirement because applying a plan always clears the published cache.
Adds both triggers as a parametrised regression asserting the retired pool's
owner observes exactly one `__aexit__` and the registry drains — not merely that
the pool is fenced — plus a mutation proof for the wait-path cleanup.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): construction-enforced async decision batches (ActivationDecisions)
The follow-up promised in the #4541 review threads. Four review rounds
traced four instances of one failure family: an async execution path
whose decision-map miss silently fell back to the synchronous
authorize() from a worker thread — the wrong API for loop-affine
providers, and under fail_closed:false a provider error there flips a
denial into an allow. The carrier was a plain dict|None held together by
caller discipline.
ActivationDecisions (deerflow/authz/activation_decisions.py) makes the
contract construction-enforced:
- a covered name returns the batched decision (unchanged semantics);
- a MISS resolves per the configured provider-error policy (fail-closed
denies, fail-open allows) with a loud WARNING — a miss is a
prepass/consumer divergence, a construction bug, not a policy
question — and never touches a provider;
- None remains the sync-chain signal (synchronous authorize() is correct
there);
- an empty candidate set still yields an empty BATCH, not None, so an
unexpected consumer name fails per policy instead of silently taking
the sync path;
- get_or_none lets publishers distinguish 'uncovered' from False;
- fail_closed travels with the batch from the resolved authorization
context.
Both middlewares (_activation_allowed in SkillActivationMiddleware and
SkillToolPolicyMiddleware) migrate to the type; their collectors return
batches; the render-decision publisher adapts. Tests 65->70+5: type unit
tests (covered/miss both directions with loud-log assertions/
get_or_none/container), middleware-level miss semantics (fail-open
allows with provider.sync_calls == [], production-default fail-closed
denies, None sync chain still synchronous), and mutation M18 (restoring
the miss's sync fallback) verified red. 252 tests green on the affected
surface.
* docs(authz): sync every decision-batch contract description to the new miss semantics
Review round 1 on #6508 — all documentation-level: the docstrings still
described the miss-as-synchronous-fallback contract this PR removes.
- _activation_allowed (both middlewares): a covered name returns the
batched decision; a miss resolves per the carried provider-error policy
with a WARNING and never touches a provider; None (sync chain /
disabled) is the only synchronous case.
- _collect_activation_decisions / _resolve_policy_registry /
_active_skills_for_paths: the reason not to diverge is now a
policy-resolved miss (denied under the production fail-closed default;
allowed without any provider consult under fail-open; logged loudly),
not a silent sync fallback.
- The two _REGISTRY_LOAD_FAILED sentinel comments, same rewording.
- Drop unused tmp_path/monkeypatch fixtures from the two new miss tests.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs(events): document durable thread seq watermark
RUN_EVENT_STREAM.md described seq as monotonic but did not cover the durable per-thread high-water mark that keeps allocations monotonic after a run is deleted. Record that behavior in the Record Envelope section, including that thread deletion drops the mark only when no events remain.
* test(scheduler): pin clock for scheduled-run timezone assertions
---------
Co-authored-by: yu1128-L <296164533+yu1128-L@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scheduler): keep a one-time task's scheduled run after an early trial
"Run once now" on a once task before its run time launched the trial as
the task's own run: the launch marked the task running and
finalize_occurrence then mapped the trial's outcome to completed, failed
or cancelled. claim_due_tasks only claims enabled tasks, so the run never
fired at run_at although next_run_at still showed it.
A manual trial launched while next_run_at is still ahead now leaves the
parent status and next_run_at unchanged, like a trial on a recurring
task, at launch, in finalize_occurrence and in the launch repair
(_associate_task_with_run). A trial after the run time has passed is
still the task's run.
* docs(changelog): reference #6512
* refactor(scheduler): scope the once-trial rule to once tasks and link its launch twin
* fix(scheduler): keep a newer due-task claim through late launch bookkeeping
With early trials leaving a once task enabled, the poller can claim the
now-due task after the trial finished but before the trial's launch
bookkeeping commits. That late update_after_launch passed can_project (the
claim had not allocated an occurrence yet), kept the claim's "running"
status under protect_terminal, and cleared the claim's lease, so the
poller's admission was rejected as stale and the task was left running
with no lease and no occurrence: unclaimable, and finalized from the
trial by stuck-once recovery.
A launch write identified by its occurrence never owns the parent lease
(scheduled admission releases the claim with the queue insert, and a
manual trial takes none), so update_after_launch now clears the lease only
for a write with no occurrence row, which is the claimer's own pre-admission
failure.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): honor sandbox.environment on the local provider
LocalSandboxProvider ignored the documented sandbox.environment config
field (only the AIO provider consumed it), leaving public generation
skills (video-generation / image-generation) with no way to receive
provider API keys on the default local sandbox: env inheritance is
scrubbed by policy (#3861), the skills declare no required-secrets, and
dashboard runs cannot supply config.context.secrets. Fixes#6461.
- LocalSandbox accepts an operator environment; execute_command layers
it through the existing authorized injection channel
(build_sandbox_env), so injected entries survive the secret scrub
while platform credentials still never leak.
- Request-scoped env (declared required-secrets) wins on key collision.
- LocalSandboxProvider resolves $VAR refs from the gateway process env,
mirroring the AIO provider, and passes the mapping to every sandbox it
yields; missing/unloadable config degrades to injecting nothing.
* fix(sandbox): address review on local sandbox.environment
- P1: operator-injected values join the bash-tool output redaction set
(request-scoped values keep precedence on name collision), so an
echoed sandbox.environment credential no longer flows back into the
model-visible tool result.
- P2: consume resolved SandboxConfig.environment values verbatim. The
file-backed AppConfig loader already substitutes $VAR references;
re-resolving corrupted credentials whose value legitimately starts
with '$'. Drop the now-unneeded AIO-style resolver.
- Validate operator environment keys at LocalSandbox construction (a
typo'd key fails at startup, not on first command execution).
- Tests: verbatim passthrough, happy-path config loading, acquire()
wiring (generic + thread), key validation, and a P1 redaction
regression through the real bash tool path.
- Docs: SandboxConfig docstring and sandbox/AGENTS.md now describe the
operator injection channel and its output masking.
* fix(sandbox): redact effective local credentials in bash results
* fix(ci): keep local sandbox guidance within inherited budget
---------
Co-authored-by: vicfei <4058491+vicfei@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
