* feat(goal): record the met goal and keep goal state on the history head
- Add a goal_outcome LastValue channel (GoalOutcomeState). Only the
guarded clear of a satisfied ordinary goal writes it, in the same
checkpoint as the clear; the reply anchor is the latest visible AI
message with text, the reply the visible signature keys on. A goal
stored without created_at still clears and gets a record.
- write_thread_goal removes the record on every other goal write and
lists it in channel_versions, new_versions and metadata writes when
it changes. Threads that never had a record keep their checkpoint
shape. The sync-only SqliteSaver and CachedHistorySaver, which take
the #6448 sync fallback, store and drop it the same way.
- goal_outcome is server-owned: POST /state and non-internal run input
that carry it get 400. A POST /state goal write, or a goal in
external run input, clears it.
- A branch head written onto its replay base resets the channels the
head lacks, so a goal or record that write_thread_goal removed does
not come back. Branching from the met turn keeps the record without
the met goal.
- The POST /history head now carries an active goal and an achieved
goal_outcome, also before any message exists, so the post-run
history refetch keeps them. Older entries stay title/thread_data.
- contracts/thread_goal_contract.json pins the chat stand-down and
check-failure codes, the record shape and status, the head keys and
the host-written reasons.
* feat(frontend): show why a goal stopped and when it was met
- The goal bar derives its state from the active goal, the goal_outcome
record and the messages: set, in progress, continuing, stopped (limit,
auto-continue off, no progress, token budget, missing evidence, failed
run, unknown code), waiting for you, waiting, couldn't check the goal,
paused and goal met. An 8/8 stand-down now reads "Stopped" with the
next step instead of "Continuing 8/8", and a count-0 stand-down no
longer looks like a fresh goal.
- "Goal met" shows while its anchor reply is the latest visible message
(a visible human message, or a visible AI message with text, moves the
chat on), so a reload agrees with the live view. An active goal wins
over the record, and this tab's /goal set or clear hides it.
- useActiveGoal drops the optimistic goal when the record of that same
goal arrives, which fixes the bar that stayed after a goal met on the
first run (the clear frame has no goal key), also for a goal stored
without created_at. Server goal values pass through isGoalState; a
/goal status read uses the backend's own active check, so the bar, the
toast and the edit lock agree with the server.
- Row 2 gives the reason and what to do next; Details holds the full
goal, the checker's own note (never host-written text, a check failure
or a paused goal's older check), the assumption note and an unknown
code, and starts collapsed in each new state. The bar is a labelled
section with a live region that announces state changes only (another
goal's same state counts) and re-reads its text after a language switch.
- The edit pencil on the latest turn is shown disabled with "Editing is
off while a goal is set" instead of hidden; the message toolbar shows
on focus-within, so keyboard users can reach it.
- A /goal set or clear rejected with 409 keeps the draft and says a run
is still going; other failures keep the server's message and the draft.
A /goal request replaced by a newer one leaves the composer alone, so
a double submit no longer empties it before the second one fails.
- Copy lives in inputBox.goalBar (en-US, zh-CN); a contract test reads
contracts/thread_goal_contract.json, and the raw-identifier guard now
catches stand-down codes and goal field names.
* test(frontend): cover the goal bar on both chat pages
- tests/e2e/goal-status.spec.ts runs every case on the chat page and the
agent chat page, since each wires the goal bar and the edit lock itself:
a goal stopped at 8/8, one waiting for details and one waiting on an
open question card show their reason and next step, also after a
reload; the locked edit pencil gets keyboard focus, shows its toolbar
and tooltip, does not open the editor, and hides while a file uploads;
Goal met after one automatic follow-up (In progress, Continuing 1/8,
then "Auto-continued once") survives a reload and hides after the next
visible message; a goal met on the first run of a new chat replaces the
set bar and frees the pencil, and a new /goal replaces Goal met while
its own run goes on; a /goal set or clear refused with 409 keeps the
draft for the retry and shows its own busy toast.
- The mock POST /history head carries only the contract's
history_head_keys, with goal only while active and goal_outcome only
when achieved; thread state and run values omit an unset goal instead
of sending goal: null. PUT and DELETE /goal drop the met record,
setGoalWritesBusy returns the Gateway's 409, and a repeat create of a
requested thread id returns the thread as is, like the Gateway.
- The raw-identifier guard also catches checker blockers and the met
record's field names.
- frontend/src/AGENTS.md: the GoalStatus sentence now names goal,
goal_outcome and useActiveGoal, reworded in place.
* test(frontend): keep the draft through a failed upload and a busy /goal
#6412 fires onSent only once a send is dispatched, so a failed upload
keeps the text, files and stored draft. A /goal the Gateway refuses with
409 never reaches sendMessage, and the composer rejects it so PromptInput
keeps the text. Drive both through the real sendMessage in one composer:
a failed upload keeps "summarize it" stored, a busy /goal keeps its text
and stored draft with no run, and the retry saves the goal and starts
one run with the objective, not the "/goal" text. After that run the
composer is empty and no stored draft comes back.
* fix(frontend): drop the stop condition's closing period before the pause clause
A stored stop condition often ends with a period, so the schedule card
read "…remain)., pauses itself" (and "…。，满足后自动暂停" in Chinese).
Trim the condition's closing sentence punctuation before the clause that
follows it. The event line, where the condition ends the sentence, is
unchanged.
* fix(frontend): point "evidence missing" at the agent, not a user reply
"Evidence missing" asked the user to reply with what's left, which is
the push a goal is meant to save. It now asks the agent to show the
result or explain what's missing, and offers /goal clear only for a goal that
no longer applies: clearing drops the met record, so work that is right
would never show "Goal met".
zh-CN follows. Copy only: no key, state or mapping changes, and the
scheduled page's reason labels are untouched.
tests/unit/components/workspace/goal-status-next-steps.test.ts checks
the row in en-US and zh-CN, and that no stop row offers /goal clear
without that condition. goal-status.dom.test.tsx and
goal-stream.dom.test.tsx now expect the new copy.
* fix(frontend): say "Couldn't check the goal" once in the goal bar
After a goal check that could not run, the bar's second line borrowed
the scheduled page's whole reason, so it repeated the chip: "Couldn't
check the goal · Couldn't check the goal; the run itself may be fine.
It's checked again after your next message." Chinese had the same
repeat.
The line now gives only why the check failed ("The run itself may be
fine", "No final reply was saved" or "The chat changed during the
check"), then when it is checked again. The screen-reader announcement
follows. The chip and the scheduled page's reason labels are unchanged.
tests/unit/components/workspace/goal-status-unchecked.test.ts checks
every check-failure code in en-US and zh-CN: the label appears once,
and the line still says why the check failed.
* feat(goal): redact the evaluator input when pii_redaction is enabled
The goal evaluator calls its model directly, outside
PiiRedactionMiddleware, so it sent raw user lines, tool arguments, tool
results and Human Input Card answers even with pii_redaction enabled.
With pii_redaction enabled:
- the messages in the evidence window are redacted whole before the
evidence caps can cut an identifier in two, as TitleMiddleware does.
This reuses redact_queued_messages, which already covers content,
call arguments and card answers. Earlier messages are left alone: the
evaluator never sees them, and the pass is synchronous work on the
event loop (0.84 s for an 800-message, 4 MB thread, 0.06 s for its
window);
- the assembled input, the goal objective included, is redacted again.
A redaction error fails the check with evaluator_failed; raw text is
never sent instead. When redaction is off, the input is byte-identical.
Thread state keeps the raw text. README.md, config.example.yaml and the
middleware docstring list the evaluator with the other redacted inputs.
tests/test_goal_pii.py puts an address where the evidence caps cut a
tool argument, a tool result and a card answer, and checks the
objective, the unchanged input when redaction is off, that only the
evidence window reaches the redactor and gives the same evidence as
redacting the whole thread, that a redaction error stands down with
evaluator_failed before the model is called, and the worker path.
* fix(goal): redact the hidden continuation when pii_redaction is enabled
The hidden /goal continuation repeats the goal objective and the
evaluator's reason and evidence summary. PiiRedactionMiddleware skips
it, since it is a framework message and not a user message, so with
pii_redaction enabled each continuation turn sent them raw to the
agent's model, and the message stayed in the thread for later model
calls. The durable-context block and the evaluator input already redact
the objective.
With pii_redaction enabled, make_goal_continuation_message now redacts
the objective, the reason and the evidence summary with redact_text
before the message is built. The thread keeps the redacted message,
which the UI hides; the goal state keeps the raw objective.
The worker builds the message before it counts the continuation. A
redaction error fails the check the way an evaluator redaction error
does: run_failed with stand_down_reason evaluator_failed, the goal stays
active, nothing is sent and the continuation budget is not spent. Only
the exception type is stored. With redaction off, the message is
byte-identical.
README.md, config.example.yaml and the middleware docstring list the
continuation with the other redacted inputs.
tests/test_goal_continuation_pii.py checks the redacted objective,
reason and evidence summary against the tokens redact_text gives, the
unchanged message with redaction off or no config, the worker path
under the run config, and a redaction error.
* docs(changelog): link the goal outcome entries to #6556
Fill the four ([#PR]) placeholders in CHANGELOG.md and CHANGELOG_zh.md with this PR's number and add the matching link references.
* fix(frontend): keep an active goal stored without timestamps visible
POST /threads/{id}/state and run input can store an active goal without
created_at or updated_at. The backend still evaluates it and rejects an
edit with 409 (is_active_goal checks only status "active"), but
isGoalState required both timestamps, so history and live values
dropped it: both chat pages hid the goal bar and unlocked the last-turn
pencil, and only a /goal status read, which used a looser predicate,
brought them back.
isGoalState now accepts any record with a string objective and status
"active", and normalizeGoalStatusRead uses it, so thread values, live
updates and status reads share one rule. Malformed or inactive values
are still dropped.
Tests cover a sparse active goal from the history head and from a live
update before the post-run refetch (bar shown, hasGoal true), and a
malformed goal that stays hidden.
* test(goal): check the outcome write metadata the same way on every saver
PostgresSaver stores checkpoint metadata through
get_serializable_checkpoint_metadata, which drops "writes", so the
[postgres] case of test_goal_outcome_round_trips_through_the_sync_fallback
failed with KeyError: 'writes' when CI ran it against a live database.
The test now records the metadata write_thread_goal passes to put and
asserts the clear's and the next goal's "writes" there, for every saver.
The stored channel values are checked as before, and on sqlite and
cached-sqlite the stored "writes" must still equal what was passed.
* docs(agents): keep goal_outcome out of the agents guidance chain
The middlewares AGENTS.md chain is already over the 98,304-byte AG002
hard limit on main, and the PR-mode check fails any chain a PR touches,
so adding goal_outcome to the ThreadState field list in
deerflow/agents/AGENTS.md failed the agent-guidance job. goal_outcome
stays documented in runtime/AGENTS.md and backend/app/gateway/AGENTS.md,
which are outside that chain.
* refactor(goal): read the evaluator's evidence and redaction from one window
format_visible_conversation recomputed the window start that the redaction
pass takes from _evidence_window. Both now use _evidence_window, so a later
change to the cap or to what counts as visible moves both.
The window test now fills the window with tool-only replies and tool results.
Before, it caught a cap change in one copy but not a change to what counts
as visible.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(community): give max_results coercion a single owner
#5865 audited every bundled provider's max_results coercion, and #5866 /
#5867 are fixing seven copies of it. Each one had to be read on its own
because four of the copies were byte-identical apart from the provider
label in their warning line.
Move the shared bar into deerflow.community.search_max_results and have
ddg_search, image_search, fastcrw and firecrawl delegate to it. Warnings
keep their existing text and their existing logger: logger is a parameter
so records keep their module attribution, which
tests/test_ddg_search_tools.py filters on.
image_search also stops spelling its own default twice -- the two tool
signatures hardcoded 5 while the same file defined DEFAULT_MAX_RESULTS.
No bound is added to any provider, and brave / groundroute / serper /
serply / sofya / tencent_wsa are left untouched so #5866 and #5867 land
without conflict. The new test pins the 13-value behaviour, the delegation
and the historical labels, and asserts that the set of providers still
hand-rolling a coercer is exactly the declared six -- so a twelfth copy
fails CI rather than being audited by hand later.
* test(search): cover generic and inline max-results coercion
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The existing cross-thread test only goes red on Python 3.13+, while every
backend CI workflow pins 3.12, so removing the yield added in #6565 would
stay green. Record a middleware event from a raw thread, join it with no
intervening await so the call_soon_threadsafe hop is provably pending, then
close(flush=True): without the yield the event is dropped on every version.
Follows up the review on #6565.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(runtime): keep thread checkpoints when a rollback cancel precedes snapshot capture
A rollback cancel that lands after try_start() flips the run to running but
before _capture_rollback_point() completes left rollback_point=None with
snapshot_capture_failed=False. The rollback finalizer treated that state as
"the thread never had a pre-run checkpoint" and called adelete_thread(thread_id),
deleting the whole conversation's checkpoints.
Initialize snapshot_capture_failed fail-closed and clear it only after the
capture completes. A completed capture that returns None still resets an empty
thread as before. Covered by two regressions in tests/test_run_worker_rollback.py
(cancellation before and during the capture await).
* fix(runtime): drop the redundant snapshot_capture_failed assignment
The fail-closed initializer owns the capture state: `snapshot_capture_failed`
is True from the top of run_agent, and the capture block runs once, so the
except branch can only observe it still True. Re-assigning it there was dead
code (review nit from @willem-bd).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): classify duration-only checkpoints without the Postgres-dropped marker
On Postgres the savers funnel metadata through langgraph's
get_serializable_checkpoint_metadata, which pops writes before the row
lands, so persist_run_history_metadata's runtime_run_duration marker never
comes back and every replay/history read path treats metadata-only duration
leaves as addressable conversation state.
is_duration_only_checkpoint now classifies in two tiers:
- writes present (memory/SQLite) is authoritative — the per-writer key is
exact, so a title leaf carrying runtime_interrupt_title with inherited
run_durations is not a duration checkpoint;
- writes absent (Postgres) falls back to the surviving stamps
(source == run_durations / run_message_ids keys).
Stamps alone are not exact: _ensure_interrupted_title and
_rollback_to_pre_run_checkpoint copy the head's metadata and inherit them.
Read paths tolerate that because every stamp-bearing leaf either copies its
parent's messages verbatim or restores an ancestor's exact message state, so
a skipped leaf only advances a message-presence scan onto a state-equivalent
checkpoint. Destructive callers opt out via stamp_fallback=False:
enforce_thread_retention keeps widening the class only through the
parent-shape check in _mark_duration_leaves_without_the_marker.
* fix(gateway): confirm the duration stamp fallback by parent shape
Review on #5988 surfaced a real false positive: write_thread_goal copies
the head's metadata and thereby inherits the run_durations/run_message_ids
stamps while changing the goal channel, and on Postgres its own writes.goal
marker is popped too — so the stamp fallback classified the goal leaf as
duration-only and the branch scan walked past it, copying channel values
from before the goal write.
The stamp fallback now also requires the duration writer's shape: it clones
the parent's channel_versions verbatim, while every state-changing leaf
(title, goal) bumps at least one version. Callers resolve the parent from
their own context — the lineage walk fetches the grandparent only for
stamp candidates, the history-window scans (branch, latest-turn,
regenerate filter, chronological fallback) index the window they already
hold — and stamps without a resolvable parent are refused: scanning one
state-equivalent copy is free, skipping a goal leaf is not.
Also corrects the mechanism claims from the first pass: langgraph's
update_state paths (rollback, compaction, manual updates) persist metadata
built from scratch and cannot inherit the stamps at all; the inheriting
writers are title and goal.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(gateway): share the stamp gate and reuse the walked grandparent
Review round 2 on #5988 — all three items, no functional changes:
- extract _has_postgres_stamps as the single gate shared by the classifier
fallback tier and the walk probe, so widening one widens the other;
- carry the grandparent fetched for a stamp candidate into the next
iteration (it is exactly the next parent), turning a chain of N Postgres
duration leaves from ~2N+1 ancestor reads into N+1;
- pin the unresolvable-grandparent promise at walk level: a stamp
candidate whose own parent link was pruned stays addressable and remains
the replay base.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(gateway): resolve persisted channel versions for the duration stamp fallback
Materialized accessor snapshots expose no checkpoint payload, so the verbatim-copy shape check always failed open and Postgres duration leaves stayed addressable. Carry the version maps alongside the snapshots (raw snapshots expose them directly, graph-backed ones re-read the raw tuple through the accessor) and use them at every replay-base scan site.
* perf(gateway): resolve stamp-fallback versions lazily and once per tuple
The Postgres stamp fallback needs the raw channel versions for its shape check, and both window scans resolved them eagerly for every entry: a branch request paid up to 2N serial roundtrips for a window of N, and the regenerate scan resolved the same tuples a third time through resolve_history_versions.
Resolve them only for stamp candidates - the marker tier and the stamp gate decide without versions - and memoize by config identity, so a marker-decided window pays nothing and each distinct tuple pays at most one raw read.
The lineage walk had the same shape: a stamp candidate the shape check refuses re-read its grandparent as the next iteration's parent, because the prefetched slot was only filled when the candidate was skipped. Ancestor reads are now memoized by config identity, so no ancestor is fetched twice.
Dropped the dead checkpointer parameter of find_checkpoint_before_message_chronologically: no caller passed it, and wiring one in from an async handler would put a blocking get_tuple on the event loop.
---------
Co-authored-by: zeng-bohan <310902929+zeng-bohan@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* fix(uploads): skip files that vanish mid-scan in list_files_in_dir
The listing iterates scandir entries and stats each survivor. A
concurrent upload delete or the staging sweep can remove a file in
between, and the resulting FileNotFoundError escapes as a 500 on the
listing endpoint. is_file() does not cover this: on POSIX it answers
from the scandir d_type without touching the filesystem again.
Catch OSError around the stat and skip the entry on ENOENT/ENOTDIR/
ELOOP, the same tolerance the chmod path just above already applies to
replacement races. Operational errors such as EACCES still propagate.
Regression tests pin both directions: a vanished entry is skipped while
the rest of the listing survives, and a permission error still raises.
Fixes#6097
* fix(uploads): log vanished entries and fix comment per review
Address willem-bd's review on #6533: the skip now emits a debug line so
"my file disappeared from the list" reports are diagnosable without
changing the response contract, and the race comment names the actual
concurrent removers (the DELETE endpoint, possibly from another
replica, and sandbox processes) instead of the staging sweep, which
cannot remove a listed non-staging file.
The directory-level race between the is_dir() guard and os.scandir()
stays out of scope; it is covered by the still-open #6096 and its PR.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(uploads): tolerate directory races during listing
* docs(uploads): scope listing guidance to avoid inherited size limits
* fix(uploads): log directory races during listing
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Since CPython gh-105836 (backported to 3.13) an awaited executor future can
complete without a loop iteration when the worker finished before the future
was chained, so a middleware event a worker handed to the owner loop with
call_soon_threadsafe may still be pending when flush() starts. The run-end
flush() -> _detach_runtime_dependencies() sequence then dropped it. Yield to
the loop once before draining and re-check _closed.
Fixes#6564
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(helm): let the sandbox provisioner run with several replicas
The chart hard-coded replicas: 1 into the provisioner Deployment, gave it
no rollout strategy and no PodDisruptionBudget, and values.yaml had no
provisioner.replicas key, so a multi-replica gateway deployment still sent
every sandbox create/discover/destroy call through one Pod: while that Pod
restarted or its node drained no sandbox could be created anywhere.
- values.yaml provisioner gains replicas (default 1, unchanged), the
gateway's surge-then-drain RollingUpdate strategy and
podDisruptionBudget {enabled: true, minAvailable: 1}.
- provisioner-deployment.yaml renders replicas and strategy from values; at
1 replica the result equals the Kubernetes default strategy.
- New provisioner-pdb.yaml, rendered only while the provisioner is enabled,
the budget is enabled and replicas > 1, selecting on the Deployment's
labels and named <fullname>-provisioner so it never collides with the
gateway budget. No multiInstance override: nothing else in the chart
renders differently per provisioner replica count.
- docker/provisioner/app.py is unchanged. It keeps no state between
requests: the labelled sandbox Pods and Services are the only registry,
create tolerates the 409 a concurrent creator produces, destroy tolerates
404, NodePorts come from the API server and there is no background loop.
The chart README and docker/provisioner/README.md record this, and
tests/test_helm_provisioner_replicas.py pins it at source level next to
the values, template and helm-render assertions.
* docs(changelog): link the provisioner replicas entry to #6543
* fix(helm): preserve percentage PodDisruptionBudget values instead of casting them to 0
Review of #6543: both PDB templates rendered minAvailable through
`| default 1 | int`, and sprig's int turns a percentage string such as
"50%" -- a value Kubernetes accepts -- into 0, a budget that permits every
Pod to be evicted. The gateway budget had the same pre-existing cast.
- New shared helper deer-flow.pdbMinAvailable in _helpers.tpl: a YAML or
--set number renders as an integer, a digit string from --set-string
becomes that integer, a percentage is preserved quoted, and 0, "0%",
more than 100%, a non-integer or anything else fails the render with a
message naming the values key and the enabled switch to use instead.
Unset still falls back to 1.
- gateway-pdb.yaml and provisioner-pdb.yaml render minAvailable through it.
- values.yaml comments, the chart README multi-replica note and the
CHANGELOG entry say integer-or-percentage and that zero is rejected.
- tests/test_helm_provisioner_replicas.py pins the helper and the two
templates at text level and adds render cases for both budgets: "50%"
with three replicas renders "50%", 2 via --set and --set-string renders
2, and abc / 0 / 0% / 150% fail naming the key.
* fix(helm): normalize json.Number before the minAvailable string branches
Review of #6543: Helm 3.18.0 loads YAML numbers as json.Number, whose
kind is "string" but which regexMatch and the other string-typed sprig
functions reject at argument validation ("expected string; got
json.Number"). deer-flow.pdbMinAvailable gated its string branch on
kindIs "string", so the unoverridden chart default minAvailable: 1 made
every gateway.replicas=2 / provisioner.replicas=2 render fail on that
release, inside the README's documented 3.8+ support window.
- The helper keeps the explicit float64/int64/int branch and now converts
everything else with toString into $text before the digit, percentage
and fail branches; regexMatch, atoi, trimSuffix and quote only ever see
that plain string. A json.Number 1 renders 1, a json.Number 1.5 still
fails (the digit regex rejects it) and the hint still prints the raw
value; bools and other kinds still fail. The helper comment records the
Helm 3.18.0 reason.
- tests/test_helm_provisioner_replicas.py pins that the string branch
runs on the toString-normalized variable, that no string-typed function
is applied to $raw and that kindIs "string" is gone, and adds an
explicit render case for both budgets with only the replica count set
(the unoverridden default must render minAvailable: 1), marked as the
Helm 3.18.0 json.Number regression guard.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The Tavily, Exa and Firecrawl tool pages and the AIO sandbox sections
tell readers to run `uv add 'deerflow-harness[tavily|exa|firecrawl|aio-sandbox]'`.
deerflow-harness has no such extras: tavily-python, exa-py, firecrawl-py
and agent-sandbox are core dependencies. uv warns about the unknown extra
and still writes it into backend/pyproject.toml.
Say that no extra dependency is required, matching the Brave and Serply
tabs and the E2B sandbox section.
* fix(channels): send the CSRF pair on IM /goal writes
`/goal <objective>` and `/goal clear` from an IM channel sent their
PUT/DELETE to the Gateway with only the internal auth headers. The
Gateway's CSRF middleware does not exempt internal auth, so with auth
enabled (the default) both writes were rejected with 403 and the channel
replied "Failed to set goal." / "Failed to clear goal.".
`_goal_request` now sends the same double-submit cookie/header pair as
the channel's LangGraph SDK client; both build it from one helper,
`ChannelManager._csrf_headers()`.
* docs(changelog): reference #6537
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
When Sandbox.connect() fails with a trusted 'sandbox not found' error,
_reconnect_live_client() used to release the deployment-level slot but
still re-raise. _evict_oldest_warm() then treated that confirmed-gone
result like an unknown-state failure: it kept the eviction tombstone and
never called _end_transition_locked(), so the local transitioning slot
leaked on every retry against an already-reaped VM. Once the leaked slots
reach the configured replicas, every acquire rejects or times out with
SANDBOX_CAPACITY_EXCEEDED while active=warm=reserved=0.
Return None from _reconnect_live_client() for trusted gone errors so all
callers (eviction, warm reclaim, discovery, reconciliation) take their
existing already-gone cleanup paths, keeping local bookkeeping consistent
with the deployment-level release on the same signal. Unknown-state
errors (network, auth) still retain the slot for a later retry.
Also report the transitioning count in SandboxCapacityExceededError
details and the wait-timeout message so hidden slot usage is visible
without a restart.
Fixes#6540
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): avoid executor starvation in file-gate waits
Keep async file-gate waiters out of the executor needed by the holder.
Preserve sync and cross-loop exclusion, and drain started file work on cancellation.
* test(agents): cover mixed file-gate callers
Document why release wakes every async waiter.
Cover a synchronous tool holder and asynchronous waiter through the middleware entry points.
* test(agents): cover sync waiters released by async file tools
Signed-off-by: LingYi <lingyi01525@gmail.com>
---------
Signed-off-by: LingYi <lingyi01525@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`_local_launcher_home()` was the last caller of `shutil.which("bash")` left
behind by #5404, so on Windows it started the WSL launcher instead of Git Bash
and all 30 tests built on that helper failed while decoding the stub's UTF-16
output. Route it through `require_script_bash()` like the other shell-script
tests.
Git Bash also answers with MSYS paths (Windows TEMP is mounted at `/tmp`), so
the home printed by the launcher could not be used from Python. Convert with
`cygpath -w` inside the shell, where values that look like shell syntax stay
intact, and keep `printf` for hosts that ship no `cygpath`.
The unquoted `~/...` case additionally needs `USERPROFILE`: Git Bash expands
`~` from `HOME`, while Python's `expanduser()` reads `USERPROFILE` on Windows.
Windows: `tests/test_support_bundle.py` goes from 30 failed / 59 passed to
89 passed. POSIX hosts keep their previous behavior.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): decode Git Bash output as UTF-8 so CJK tools are not garbled
Git Bash/MSYS always writes UTF-8 to its pipes, but the MSYS branch of
LocalSandbox.execute_command called _run_windows_command without an
encoding, so the capture fell back to locale.getpreferredencoding(). On a
non-UTF-8 host (GBK on zh-CN Windows) that decodes UTF-8 bytes with the host
code page and mangles every non-ASCII tool result. The pipe decoder replaces
instead of raising, so the corruption is silent. The PowerShell branch right
next to it has pinned UTF-8 since #5440.
Decode the MSYS branch as UTF-8 too, and pin the behaviour in
tests/test_local_sandbox_encoding.py: a unit test for the decoder choice and
a live round-trip through a real Git Bash (skipped when it is absent).
* fix(sandbox): keep locale fallback for native-child output under Git Bash
Git Bash/MSYS writes its own output as UTF-8, but Windows-native children
(python.exe, CLI shims) inherit the pipe and keep writing the host code
page. Forcing UTF-8 with errors=replace turned that output into U+FFFD.
Decode the captured buffer as UTF-8 first and retry with the locale when it
is not valid UTF-8, so both MSYS UTF-8 output and native-child code-page
output survive. The PowerShell branch is unchanged (its own output is pinned
to UTF-8 by the preamble); the fallback is only wired on the MSYS path.
Adds native-Python CJK regressions through execute_command.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(auth): key the compose login lockout by client, not by nginx
Failed logins are counted per client IP, and the Gateway honors X-Real-IP
only from a peer listed in AUTH_TRUSTED_PROXIES. The compose files never
set it, and every browser request reaches the Gateway from the nginx
container, so all logins shared nginx's address: five wrong passwords
from anyone locked every user out for five minutes.
AUTH_TRUSTED_PROXIES now also accepts hostnames, resolved off the event
loop and cached for 10 seconds (failed lookups included, and only when
X-Real-IP is present), and both compose files default it to the bundled
nginx service, whose address Docker assigns. nginx overwrites X-Real-IP
with $remote_addr on every Gateway route; a test pins that for the three
maintained configs.
* docs(changelog): reference #6519
* fix(auth): share one trusted-proxy lookup across concurrent cache misses
A burst of setup-status / login requests on an expired hostname entry
started one getaddrinfo per request while DNS was slow, occupying the
default executor that password verification also uses. The cache check
and an in-flight task registration now happen without a yield between
them; later callers join that task through asyncio.shield, so a cancelled
request cannot cancel the lookup the others joined, and the task caches
its own result.
The compose default keeps `:-` on purpose: every browser request arrives
through nginx, so an empty AUTH_TRUSTED_PROXIES would only bring back the
shared lockout. The compose comment and .env.example say so, and the
rendered-compose test now covers the explicitly empty value.
* fix(docker): keep a .env AUTH_TRUSTED_PROXIES under make docker-start
docker.sh start runs Compose from docker/ without --env-file, so the dev
compose file's ${AUTH_TRUSTED_PROXIES:-nginx} interpolated from the shell
only, and the resolved environment entry outranked the operator's value
loaded through env_file. Export it from .env alongside the proxy variables
(a shell export still wins), and strip a CRLF before the quotes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The two Windows-only cases in tests/test_mcp_file_migration.py asserted a
literal "file:///C:/" / "file://C:" prefix on a path built from the pytest
temp dir. On a host whose checkout or temp dir lives on another drive both
assertions fail, so the only coverage of the drive-qualified file-URI forms
handled by _local_uri_to_virtual_path and _rewrite_local_paths_in_text never
ran there.
Derive the expected drive from the fixture path instead. The URI forms under
test are unchanged, so coverage holds on any drive.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): selectively reconcile changed deployment servers
* fix(mcp): validate the frozen task snapshot before installing epochs
Durable background calls resolve their connection from the configuration frozen
at Gateway startup, so installing a newer revision's binding epoch fenced every
status/cancel call with `StaleMCPBindingError` — even though hot reload of that
server is correctly rejected. The rejection only landed in `get_mcp_tools()`,
which runs after the cache had already installed the epoch.
Both decisions that install an epoch now consult the frozen snapshot first. The
reconciliation planner falls back to the conservative full reset, where callers
that still run against the startup configuration rebind in the replacement pool,
and never installs the rejected epoch. A discovery claim raises the same
`McpTaskConfigurationError` that `get_mcp_tools()` would, before touching the
pool and without recording the revision as applied. Unrelated non-task servers
still reconcile selectively.
Adds the three requested regressions — a rejected hot change keeps background
task callers working, a rejected revision installs neither epoch nor tombstone
(changed connection and disabled server), and an unrelated server change stays
selective — plus mutation proofs for both new gates.
* fix(mcp): always tear down a pool the claim path retired
`reset_session_pool()` only fences and unlinks the singleton; the owners are
signalled by `close_all_sync()`. The claim ran that teardown *after* its critical
section, so a claim that retired the pool for a new shared-reset generation and
then raised — a revision the frozen durable-task snapshot rejects is the
reachable case — left the retired pool's sessions running without ever receiving
a close signal.
Wrap the claim in `try`/`finally` so the retired pool is torn down on every path,
still outside every lock and after the lock is released. Adds a regression
covering the shared-reset-then-rejected-revision sequence: the error propagates,
the retired pool's owner observes exactly one `__aexit__`, the replacement pool
holds no rejected epoch, and restoring the startup configuration initializes
normally again.
* fix(mcp): deliver retired-pool cleanup before the cache reader waits
A synchronous `get_cached_mcp_tools()` reader can apply a full reconciliation
while a discovery is in flight — an interceptor change and a new shared-reset
generation both qualify, because the applied baseline exists before any cache is
published. `reset_session_pool()` only fences and unlinks the pool, and the
reader then entered its wait/retry branch from inside the condition, skipping the
`close_all_sync()` that signals the retired pool's owners. The replacement cache
published normally while the discarded pool kept a live owner and stdio
subprocess that no later reset could reach.
Hoist the observed waiting generation out of the critical section and deliver the
retirement before every wait/retry path, then re-acquire the condition and wait
on that exact generation so the predicate cannot miss a wakeup. Cleanup stays
outside `_init_condition`, and the early `return _mcp_tools_cache` cannot skip a
retirement because applying a plan always clears the published cache.
Adds both triggers as a parametrised regression asserting the retired pool's
owner observes exactly one `__aexit__` and the registry drains — not merely that
the pool is fenced — plus a mutation proof for the wait-path cleanup.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): construction-enforced async decision batches (ActivationDecisions)
The follow-up promised in the #4541 review threads. Four review rounds
traced four instances of one failure family: an async execution path
whose decision-map miss silently fell back to the synchronous
authorize() from a worker thread — the wrong API for loop-affine
providers, and under fail_closed:false a provider error there flips a
denial into an allow. The carrier was a plain dict|None held together by
caller discipline.
ActivationDecisions (deerflow/authz/activation_decisions.py) makes the
contract construction-enforced:
- a covered name returns the batched decision (unchanged semantics);
- a MISS resolves per the configured provider-error policy (fail-closed
denies, fail-open allows) with a loud WARNING — a miss is a
prepass/consumer divergence, a construction bug, not a policy
question — and never touches a provider;
- None remains the sync-chain signal (synchronous authorize() is correct
there);
- an empty candidate set still yields an empty BATCH, not None, so an
unexpected consumer name fails per policy instead of silently taking
the sync path;
- get_or_none lets publishers distinguish 'uncovered' from False;
- fail_closed travels with the batch from the resolved authorization
context.
Both middlewares (_activation_allowed in SkillActivationMiddleware and
SkillToolPolicyMiddleware) migrate to the type; their collectors return
batches; the render-decision publisher adapts. Tests 65->70+5: type unit
tests (covered/miss both directions with loud-log assertions/
get_or_none/container), middleware-level miss semantics (fail-open
allows with provider.sync_calls == [], production-default fail-closed
denies, None sync chain still synchronous), and mutation M18 (restoring
the miss's sync fallback) verified red. 252 tests green on the affected
surface.
* docs(authz): sync every decision-batch contract description to the new miss semantics
Review round 1 on #6508 — all documentation-level: the docstrings still
described the miss-as-synchronous-fallback contract this PR removes.
- _activation_allowed (both middlewares): a covered name returns the
batched decision; a miss resolves per the carried provider-error policy
with a WARNING and never touches a provider; None (sync chain /
disabled) is the only synchronous case.
- _collect_activation_decisions / _resolve_policy_registry /
_active_skills_for_paths: the reason not to diverge is now a
policy-resolved miss (denied under the production fail-closed default;
allowed without any provider consult under fail-open; logged loudly),
not a silent sync fallback.
- The two _REGISTRY_LOAD_FAILED sentinel comments, same rewording.
- Drop unused tmp_path/monkeypatch fixtures from the two new miss tests.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs(events): document durable thread seq watermark
RUN_EVENT_STREAM.md described seq as monotonic but did not cover the durable per-thread high-water mark that keeps allocations monotonic after a run is deleted. Record that behavior in the Record Envelope section, including that thread deletion drops the mark only when no events remain.
* test(scheduler): pin clock for scheduled-run timezone assertions
---------
Co-authored-by: yu1128-L <296164533+yu1128-L@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scheduler): keep a one-time task's scheduled run after an early trial
"Run once now" on a once task before its run time launched the trial as
the task's own run: the launch marked the task running and
finalize_occurrence then mapped the trial's outcome to completed, failed
or cancelled. claim_due_tasks only claims enabled tasks, so the run never
fired at run_at although next_run_at still showed it.
A manual trial launched while next_run_at is still ahead now leaves the
parent status and next_run_at unchanged, like a trial on a recurring
task, at launch, in finalize_occurrence and in the launch repair
(_associate_task_with_run). A trial after the run time has passed is
still the task's run.
* docs(changelog): reference #6512
* refactor(scheduler): scope the once-trial rule to once tasks and link its launch twin
* fix(scheduler): keep a newer due-task claim through late launch bookkeeping
With early trials leaving a once task enabled, the poller can claim the
now-due task after the trial finished but before the trial's launch
bookkeeping commits. That late update_after_launch passed can_project (the
claim had not allocated an occurrence yet), kept the claim's "running"
status under protect_terminal, and cleared the claim's lease, so the
poller's admission was rejected as stale and the task was left running
with no lease and no occurrence: unclaimable, and finalized from the
trial by stuck-once recovery.
A launch write identified by its occurrence never owns the parent lease
(scheduled admission releases the claim with the queue insert, and a
manual trial takes none), so update_after_launch now clears the lease only
for a write with no occurrence row, which is the claimer's own pre-admission
failure.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): honor sandbox.environment on the local provider
LocalSandboxProvider ignored the documented sandbox.environment config
field (only the AIO provider consumed it), leaving public generation
skills (video-generation / image-generation) with no way to receive
provider API keys on the default local sandbox: env inheritance is
scrubbed by policy (#3861), the skills declare no required-secrets, and
dashboard runs cannot supply config.context.secrets. Fixes#6461.
- LocalSandbox accepts an operator environment; execute_command layers
it through the existing authorized injection channel
(build_sandbox_env), so injected entries survive the secret scrub
while platform credentials still never leak.
- Request-scoped env (declared required-secrets) wins on key collision.
- LocalSandboxProvider resolves $VAR refs from the gateway process env,
mirroring the AIO provider, and passes the mapping to every sandbox it
yields; missing/unloadable config degrades to injecting nothing.
* fix(sandbox): address review on local sandbox.environment
- P1: operator-injected values join the bash-tool output redaction set
(request-scoped values keep precedence on name collision), so an
echoed sandbox.environment credential no longer flows back into the
model-visible tool result.
- P2: consume resolved SandboxConfig.environment values verbatim. The
file-backed AppConfig loader already substitutes $VAR references;
re-resolving corrupted credentials whose value legitimately starts
with '$'. Drop the now-unneeded AIO-style resolver.
- Validate operator environment keys at LocalSandbox construction (a
typo'd key fails at startup, not on first command execution).
- Tests: verbatim passthrough, happy-path config loading, acquire()
wiring (generic + thread), key validation, and a P1 redaction
regression through the real bash tool path.
- Docs: SandboxConfig docstring and sandbox/AGENTS.md now describe the
operator injection channel and its output masking.
* fix(sandbox): redact effective local credentials in bash results
* fix(ci): keep local sandbox guidance within inherited budget
---------
Co-authored-by: vicfei <4058491+vicfei@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(auth): count login lockouts in the database across Gateway replicas
POST /api/v1/auth/login/local throttled failed logins per client IP in a
module-level dict, so with N Gateway replicas behind one load balancer an
attacker got N x auth.local.max_login_attempts guesses and a lockout on
one replica was invisible to the others.
Move the counter behind a LoginThrottleStore (deerflow.persistence.
login_throttle): MemoryLoginThrottleStore keeps the historical per-process
semantics and 10k tracked-IP bound; SqlLoginThrottleStore shares one
login_throttle row per IP through the application database, incrementing
and starting the lock in a single dialect-native upsert, applying check
decisions with compare-and-set predicates in a separate write
transaction, and sweeping served locks plus 24h-idle counters in bounded
batches. Both stores pin the existing contract: the lock starts when the
count reaches max_login_attempts, the committed duration is honored when
the policy shrinks mid-lock, a served sentence is never resurrected,
tightening keeps the count, success resets the IP, and the 429 status and
message are unchanged.
New startup-only selector auth.local.throttle_storage (auto | memory | db,
default auto = the database whenever database.backend is sqlite or
postgres). The Gateway installs the store once at startup after the
persistence engine; deps._validate_login_throttle_storage warns when a
multi-process deployment keeps memory. Migration 0034_login_throttle
creates the table; config_version 55 -> 56 in config.example.yaml and the
Helm chart. Docs: README, CONFIGURATION.md, AUTH_DESIGN.md, config and
migrations AGENTS.md, CHANGELOG.
* docs(changelog): link the login throttle entry to #6501
* fix(auth): re-check expiry on the row the login-throttle sweep deletes
The bounded cleanup in SqlLoginThrottleStore.record_failure deleted
`WHERE ip IN (SELECT ip ... WHERE <served or stale> LIMIT 200)`. On
PostgreSQL READ COMMITTED the expiry predicate applied only inside the
snapshot-bound subquery: when the DELETE had to wait on a row lock held
by a concurrent check() that was extending the same lock under a raised
lockout_seconds, PostgreSQL re-evaluated only the outer WHERE on the
updated tuple, IP membership still held for the old candidate, and the
now-active lock was deleted; the next attempt from that IP got a clean
record and reached password verification (review of #6501, P2).
Repeat the served/stale predicate on the DELETE target as well
(SqlLoginThrottleStore.sweep_statement), keeping the bounded candidate
subquery, so the re-evaluation sees the updated row's expiry. The other
writes already carry their full compare-and-set predicate on the target
row: _discard and the check() UPDATE both use _matches (ip, fail_count,
locked_at, lock_duration_seconds), so a tuple changed while they waited
no longer matches and the stale decision is skipped. _discard keeps the
equality compare-and-set rather than adding an expiry condition: its
early-release branch deletes a lock whose committed sentence has not
elapsed under the lowered live policy, which an expiry predicate would
wrongly block, and the equality predicate already pins every input of
the decision. reset stays unconditional by design.
Tests: a dialect-compile test asserts the DELETE's own WHERE carries the
expiry predicate for postgresql and sqlite (red on the previous HEAD), and
a PostgreSQL-only test (skipped without TEST_POSTGRES_URI) reproduces the
update-versus-sweep interleaving with two engines, waiting on
pg_stat_activity for the sweep to block before the extender commits.
Docs: sql.py concurrency contract and the migrations guide's 0034 bullet.
* fix(auth): keep an active login lock's clock and fail closed on explicit db
Address the "changes requested" review on #6501.
Lock clock (confirmed defect): record_failure re-stamped locked_at = now
for every failure on an already-locked row, so a sustained attacker's
sentence slid forward with each attempt. Both stores now apply one rule
in one atomic decision: new_count = fail_count + 1; below max_attempts the
IP counts with no lock (a failure after the operator raised the threshold
clears an existing lock); at or over it, a failure during an *active* lock
keeps locked_at and the committed lock_duration unchanged, while a
never-locked or served row starts anew with now / lockout_seconds (so the
failure that reaches a raised threshold starts a fresh lock rather than
keeping a stale locked_at). The SQL upsert expresses it as nested case()
on the row's own values; the memory store mirrors it. The sweeps in both
stores now leave the row being recorded to that decision instead of
deleting a served lock or stale counter first, which split the contract
(SQL would have produced a fresh (1, NULL, NULL) row). base.py states the
rule.
Explicit db downgrade: resolve_login_throttle_store raises SystemExit —
the failure type of the other startup validations in app.gateway.deps —
when throttle_storage='db' resolves to the shared table but no engine is
initialised, so langgraph_runtime refuses to start instead of silently
running every replica on per-process counters; the lazy bare-app path
applies the same rule and installs nothing. 'auto' keeps warning and
falling back. 'db' on a memory database still degrades with a warning
(there is no shared table to insist on).
Non-blocking items: _key() logs a warning when a client key is truncated
to 255 characters (never the key itself); the sql.py docstring covers
wall-clock locked_at across replicas (run NTP; skew shifts evaluation by
that much) and why STALE_COUNTER_SECONDS=24h / SWEEP_BATCH_SIZE=200 were
chosen; a store-level blocking-I/O anchor covers the SQL reset path; the
_validate_login_throttle_storage warning was verified to render
'throttle_storage=memory' for the StrEnum (the gate test now passes the
enum member and asserts that text). reset stays unconditional: a peer
failure that commits after a reset leaves fail_count=1, one genuine
failure after the cleared history, which a new test documents.
Docs: CHANGELOG entry, config.example.yaml, CONFIGURATION.md,
AUTH_DESIGN.md and the throttle_storage field description.
* fix(auth): re-evaluate a login-throttle check whose compare-and-set missed
SqlLoginThrottleStore.check applied its decision (commit the live
duration, or clear a served / released lock) with a compare-and-set
predicate on the snapshot it had read, but ignored the affected-row
count. When a peer changed the row between the read and the write — an
admitted wrong password incrementing fail_count on another replica, or a
successful login clearing the row — the UPDATE or DELETE matched zero
rows, yet check reported the decision as applied: it could return an
extended 60s sentence while the row kept its 1s duration, so the next
check cleared the lock and let password verification through; a live
duration decrease could be dropped the same way; and a row a peer had
just cleared was still reported as locked (#6501 review, P2).
Every compare-and-set write in check now checks result.rowcount. On a
miss the row is re-read and re-evaluated from the top, up to
CHECK_CAS_ATTEMPTS (3) times, applying a fresh compare-and-set decision
on the new snapshot and never clobbering a reset or replacement: a row
that is gone is reported as allowed, a fresh peer lock is reported as
locked, and an incremented count gets the extension or decrease
committed on it. Exhausting the bound fails closed: nothing is written
and the answer is the last snapshot's lock under the live policy, logged
at debug. The read stays in its own short transaction and each write in
a separate one.
The evaluation is now one shared pure function, evaluate_check in
base.py (CheckDecision), used by both stores so the contract cannot
diverge. The memory store keeps a single pass: its compare-and-set
cannot miss because there is no await between the read and the write on
the loop thread, which its docstring now states.
Tests (red first on df26c2acf) use a session proxy that runs the peer
mutation on a second engine right after the snapshot read ends its
transaction: the reviewer's raised-duration interleaving (59.5s reported
and committed, the increment survives, a later check still locked), the
mirrored decrease, early release on the fresh row, a reset landing
between read and write (0.0, no row recreated), and the bound-exhausted
fail-closed case.
* docs(changelog): resolve merge markers left in the Unreleased section
* test(migrations): pin 0035_login_throttle as the head in the 0034 watermark test
Upstream #6477 added test_migration_0034_run_event_seq_watermark.py asserting
0034 is the single head; this branch chains 0035_login_throttle after it, so
the test now pins the current head and walks the chain like its siblings.
* perf(auth): one session per login-throttle check, one transaction per failure
Two non-blocking performance suggestions from the #6501 review.
Single session per check. The router probed the row with store.get()
and then called store.check(), which re-read it: a recorded IP cost two
connection-pool checkouts and two SELECTs per request on the
unauthenticated login path, letting a lockout flood compete with
legitimate traffic for Postgres connections. LoginThrottleStore.check
now takes the policy as an async callable (LoginThrottlePolicy) and
resolves it lazily inside the store: the probe, the policy resolution
and the compare-and-set loop share one session, a clean IP returns 0.0
without ever calling the policy (the config.yaml read is still skipped
for the common case), and a recorded IP resolves it exactly once, also
across compare-and-set retries. The decision is made on a snapshot read
after the resolution: the policy read is a yield point, and a decision
that needs no write has no compare-and-set to catch a row a peer changed
meanwhile — so one session, but still a read after the yield. The
memory store takes the same signature and re-reads its record after the
await for the same reason. The router passes its asyncio.to_thread
policy read as the callable and drops the separate probe; get stays on
the protocol for tests and anchors. No short-TTL verdict cache: it would
mask a successful login's reset for the TTL window, and the single
session already halves the hot path.
Sweep in the upsert's transaction, plus an index. record_failure ran the
housekeeping DELETE in its own transaction and the upsert in a second
one, two commits (two WAL flushes on PostgreSQL) per failed login; the
sweep now rides in the upsert's transaction with one commit, the keep
exclusion already protecting the row being recorded. The new
ix_login_throttle_updated_at (ORM __table_args__ and the unreleased
0035 migration, created after the table and dropped before it, each
guarded separately so a table left by an interrupted attempt still gets
its index) serves the sweep's stale-counter predicate; the served-lock
predicate scans a table the sweep itself keeps small.
Tests (red first on a137fbc47): a clean IP never resolves the policy and
a recorded IP resolves it exactly once (both stores), the SQL store opens
exactly one session per check and commits once per record_failure
(counting wrappers), the migration test asserts the index after upgrade,
after completing a partial upgrade that lacked it, and gone after
downgrade while compare_metadata stays empty; the router-level
"mutated during policy resolution" tests now inject their mutation
through the policy callable.
* docs(changelog): note config_version 57 in the login throttle entry
* fix(channels): fence restart during service shutdown
* test(channels): fence restart during service stop
* test(channels): exercise restart after shutdown channel snapshot
* docs(channels): document service shutdown publication fence
* fix(channels): keep agent guidance within CI hard budget
* fix(channels): fence publication after shutdown completes
* test(channels): cover resume after completed service shutdown
* test(channels): mark direct-start fixtures as active services
* test(channels): mark direct-start fixtures as active services
* style(channels): format shutdown regression test
* fix(channels): drain channel startup completing after service stop
* test(channels): ensure late channel startup is drained after stop
* style(tests): format channel shutdown regression
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scanner): cache decoded text atomically
_cache_text_file keyed the cache on the virtual path and returned the path, and
the caller serves that path as the file's content, so a write that failed
part-way left a truncated entry that every later read handed back as the preview.
Publish through a temporary file plus os.replace.
* test(scanner): pin the cache entry against a failed publish
The helper-level case cannot distinguish the call site, so this drives the real
entry point with a failing publish and asserts the cached entry is untouched.
* docs(sandbox): keep agent guidance within its size limit
* fix(scanner): preserve publish errors during cleanup
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers
DeerMem's derived SQLite FTS5 index for every user was one WAL database pinned
to {storage_path}/.retrieval. With several Gateway Pods sharing one home volume
each Pod opened that file over a network filesystem (unsupported by SQLite),
every Pod start emptied and refilled it under its peers, one Pod's corruption
recovery deleted files the others held open, and a Pod never learned that a
peer had written a user's facts, so memory_search served stale results.
- New backend-private key memory.backend_config.retrieval_index_path: empty
keeps {storage_path}/.retrieval, a relative path resolves against
storage_path, an absolute path is used as-is. The full rebuild and the
corruption recovery act only on that configured index.
- rebuild_index records each agent scope's manifest signature
(mtime_ns, size, revision) before reading its facts; search_facts rebuilds a
scope whose live signature differs because another process wrote the user's
memory. An own commit advances the recorded signatures of that user's scopes
only when they were in sync at the previous revision, so own writes never
rebuild while an interleaved peer write still does.
- Gateway startup warns, in the style of _validate_agent_storage, when a
declared multi-instance deployment keeps the DeerMem index inside
storage_path.
- config.example.yaml documents the key; config_version 55 -> 56, mirrored in
the Helm chart and its README.
- Helm mounts an emptyDir at /var/lib/deerflow/memory-index, points the key at
it, drops the legacy memory.storage_path: memory.json line, and documents the
memory layout next to the other multi-replica prerequisites.
* docs(changelog): link the memory retrieval index entry to #6494
* fix(memory): never advance the index signature on a no-op commit and resync whole scopes
Two review findings on the cross-replica retrieval re-sync:
- save() and apply_changes() inferred the replaced manifest revision as
"returned revision - 1". The commit helper returns the existing manifest
unchanged for a no-op (unchanged-value or empty change set), so after a peer
bumped the manifest an own no-op write advanced this process's stale index
signature to the peer's revision without carrying the peer's delta: a fact the
peer deleted stayed searchable until the manifest changed again. Both paths
now read the real pre-commit revision under the user lock and pass
committed=(pre_revision, signature) only when the commit produced a new
revision; a no-op passes None so the next search still rebuilds the scope.
- The scoped rebuild (signature mismatch and the pre-existing dirty-scope path)
listed facts through list_facts() with its default limit of 100 and then
replaced every row of the scope, so a scope with more than 100 facts lost its
tail on the first re-sync. It now reads the scope's complete fact list.
* fix(memory): fence index signature promotion against a concurrent scope rebuild
Review finding on the cross-replica re-sync: a successful adapter notification
does not prove its delta is still in the index when the signature is promoted.
The SQLite lock covers single adapter calls only, and neither the scope lock
nor DeerMem's warm-up lock spans "notify, then promote", so rebuild_index can
replace a scope's rows in between. After a peer summary-only commit (r+1), A's
search starts a scoped refresh that captures the r+1 signature and reads f1/f2;
A's own deletion of f2 commits r+2 and removes the row; the refresh then
installs its stale f1/f2 and marks the scope synced at r+1; the deletion
resumes and promoted that signature to r+2, so later searches saw the live
signature as equal and kept returning the resurrected f2 until another manifest
change.
Mechanism: a generation fence rather than a lock. rebuild_index bumps a
per-scope generation (scoped rebuild) or a storage-wide generation (full
rebuild) under _cache_lock whenever it publishes or forgets a scope's rows, in
both mark_synced and forget_synced. _dispatch_retrieval_notifications snapshots
the generations under _cache_lock before its first adapter call and, at
promotion time, skips every scope whose generation moved (and everything when
the storage-wide generation moved), leaving the rebuild's own captured
signature in place; that signature predates the commit, so the next search's
comparison rebuilds the scope. A lock would have had to serialize notification
plus promotion with bulk install plus publication while search_facts ->
rebuild_index -> load() already takes the scope, file and cache locks; the
fence adds no lock-order obligations and only costs a redundant rebuild in the
rare race. Every interleaving is covered: a rebuild that completes before the
snapshot is reflected in the captured generation and promotion proceeds on an
index that carries both the rebuild's rows and this write's delta; one that
publishes between snapshot and promotion is skipped; one that publishes after
the promotion records its stale captured signature, which the next search
catches.
Tests reproduce both orders deterministically with threading.Event ordering
(adapter wrapper that holds the bulk install and the remove notification):
test_promotion_is_fenced_against_a_refresh_that_installs_in_between (red before
the fix) and
test_stale_refresh_installing_after_a_promotion_is_caught_by_the_next_search
(regression guard for the mirrored order).
* fix(memory): publish a rebuild's signature under the lock that replaced its rows
Review finding on the cross-replica re-sync: two peer-triggered refreshes of
one scope could replace rows in one order and publish their captured
signatures in the other. On a warmed Pod A, refresh R1 captures r+1 with f1/f2
and waits before installing; a peer deletes f2 at r+2; refresh R2 captures
r+2, installs f1 only and pauses after bulk_rebuild returns, before
mark_synced; R1 installs its stale f1/f2 and publishes r+1; R2 then publishes
r+2 over R1's rows. Later searches saw the live signature as equal and kept
returning the deleted fact. The generation fence only guards own-write
promotion, and the adapter's SQLite lock ends when the install returns.
Mechanism: a per-storage _retrieval_publish_lock held across the row
replacement and the signature publication, for the bulk path (bulk_rebuild +
mark_synced / forget_synced) and the clear+upsert fallback, scoped and full
alike. Overlapping refreshes of a scope therefore serialize their
install+publish pairs: whichever installs last also publishes last, and a stale
captured signature published last differs from the live manifest, so the next
search's compare rebuilds the scope. Fact reading (load() / the glob) stays
outside the lock so a slow read never blocks refreshes of other scopes. The
startup full rebuild holds the lock for its single bulk install; scoped
refreshes wait for it, which is acceptable because before warm-up completes a
search would be rebuilding its scope anyway (DeerMem.warm_retrieval takes its
own _retrieval_lock only after rebuild() returns, so the two never nest).
Lock order: the publish lock is acquired with no storage lock held --
search_facts releases _cache_lock before calling rebuild_index, reload() leaves
its scope/file-lock block before rebuilding, and DeerMem._ensure_retrieval_scopes
holds only its outer RLock, which nothing inside rebuild_index acquires. Inside
the lock only the adapter's own lock and _cache_lock are taken, and both are
leaf locks that never call back into storage, so no cycle is possible.
Test: test_rebuild_publication_stays_ordered_with_row_replacement drives the
reviewer's interleaving with threading.Event ordering (red before the fix). The
pause harness now holds a refresh after its fact read, outside the publish
lock, instead of inside the adapter's rebuild, so the earlier fence tests keep
their semantics without holding the new lock while paused.
* fix(memory): read each manifest once per full rebuild and serve the local index on a compare read error
Two suggestion-level review items on the cross-replica retrieval re-sync.
- The full rebuild cached manifest signatures keyed by (manifest path, agent
name), but _scope_signature reads only the manifest file -- every agent
bucket of a user shares one memory.json -- so the startup scan stat+read+
parsed the same file once per bucket that has facts. The cache is now keyed
by the manifest path alone while synced signatures are still recorded per
(user, agent) scope: users manifest reads instead of users x agents.
- The per-search freshness compare put a shared-volume manifest read on every
search_facts call, and _load_memory_file raises MemoryStorageCorruption on
OSError and malformed JSON, so a transient network-filesystem read error or a
corrupt memory.json failed a search the Pod-local index could still answer.
The compare now catches OSError, ValueError and MemoryStorageCorruption: it
logs a warning naming the scope (traceback at debug), serves the current
local index for that scope and leaves the recorded synced signature
untouched so the next search compares again. It deliberately does not mark
the scope dirty: a scoped rebuild would hit the same failing read, forget the
scope's signature and fail the search path too.
* docs(changelog): resolve merge markers left in the Unreleased section
* fix(memory): forget a scope's index signature when an own delta cannot be proved compatible
Review finding on the cross-replica re-sync: skipping promotion left a newer
rebuild's signature trusted even after this dispatcher had overwritten its rows
with an older notification. On a warmed Pod A, an own PATCH of f2 commits r+1
and pauses after releasing the user lock, before its adapter upsert; Pod B
deletes f2 at r+2; a concurrent search on A detects the peer change, rebuilds
the scope without f2 and publishes r+2; the pending r+1 upsert then reinserts
the deleted fact, and the revision check merely skipped because the synced
revision was r+2 rather than the PATCH's previous r. The signature stayed equal
to the live manifest, so searches returned the deleted fact indefinitely. The
generation fence could not help: it only suppresses promotion, the stale
adapter mutation still happens.
Rule: after an own commit's notifications are applied, the mutated scope's
recorded signature may stay only when the delta is provably compatible with
the published snapshot -- the scope is synced at exactly previous_revision and
no rebuild published rows for it (or storage-wide) since the fence snapshot --
in which case it is promoted as before. In every other case where this
dispatcher mutated the adapter for the scope (synced at another revision,
newer or older; generation moved; not synced at all; or notifications without
a committed revision, i.e. migration and clear paths) the scope's signature is
forgotten so the next search's compare rebuilds it. The user's other scopes
were not mutated by this delta: they are promoted when provable and otherwise
left untouched. The decision runs under _cache_lock together with the
generation reads, so a rebuild publishing concurrently is either observed or
observes the forget; the failure path (dirty + forget) is unchanged.
Orders covered: the reviewer's (publish r+2, then the stale upsert) forgets and
the next search rebuilds without f2; the earlier fence order (publish between
adapter op and promotion) now forgets too, one redundant rebuild; no concurrent
rebuild promotes exactly as before.
Test: test_stale_own_delta_replayed_over_a_newer_snapshot_forgets_the_scope
holds the PATCH's adapter upsert after the user lock is released (new
pause_next_upsert hook on the test adapter), Event-ordered, red before the fix.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(browserless): send the web_fetch token as a query parameter
fetch_html_with_status put the configured token in the /content JSON body.
Browserless reads the token only from the token query parameter or the
Authorization header, and checks it before reading the body, so with
BROWSERLESS_TOKEN set every web_fetch failed with HTTP 401; against an
instance started without TOKEN the unknown body key failed schema
validation with HTTP 400. capture_screenshot already sent ?token=, so
web_capture worked with the same token.
Both endpoints now take the token from BrowserlessClient._auth_params().
The test that pinned the body token is rewritten, and a stand-in for the
Browserless request gate (query/header auth, no extra body keys) drives
both endpoints through the real httpx request path.
* docs(changelog): reference #6484
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(events): preserve thread seq watermark across run deletion
* fix(events): clean up watermarks for empty threads
* perf(events): avoid duplicate fence on run deletion
delete_by_run already reaches _max_seq_for_thread, which acquires the transaction-scoped thread mutation fence. Keep the count and delete under that same transaction lock without taking the Postgres advisory lock twice, and pin the single acquisition in the existing test.
---------
Co-authored-by: yu1128-L <296164533+yu1128-L@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): fail fast when DEER_FLOW_HOME is not writable
`make docker-start` bind-mounts the host `backend/` directory into the gateway
container, whose process runs as root, so any state it creates under
`backend/.deer-flow` ends up owned by root on the host. A later `make up` runs
`scripts/deploy.sh` as the invoking user and only fails deep into the run with a
bare "Permission denied" while persisting generated secrets (deploy.sh:234).
Check writability right after `mkdir -p "$DEER_FLOW_HOME"` and, when it fails,
print the exact `chown` recovery command instead of letting the run get halfway
through. `make down` is excluded so stopping the stack is never blocked.
Fixes#6462
* test(deploy): cover runtime home writability preflight
* fix(deploy): diagnose persisted secret permissions and preserve teardown
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
