* fix(auth): count login lockouts in the database across Gateway replicas
POST /api/v1/auth/login/local throttled failed logins per client IP in a
module-level dict, so with N Gateway replicas behind one load balancer an
attacker got N x auth.local.max_login_attempts guesses and a lockout on
one replica was invisible to the others.
Move the counter behind a LoginThrottleStore (deerflow.persistence.
login_throttle): MemoryLoginThrottleStore keeps the historical per-process
semantics and 10k tracked-IP bound; SqlLoginThrottleStore shares one
login_throttle row per IP through the application database, incrementing
and starting the lock in a single dialect-native upsert, applying check
decisions with compare-and-set predicates in a separate write
transaction, and sweeping served locks plus 24h-idle counters in bounded
batches. Both stores pin the existing contract: the lock starts when the
count reaches max_login_attempts, the committed duration is honored when
the policy shrinks mid-lock, a served sentence is never resurrected,
tightening keeps the count, success resets the IP, and the 429 status and
message are unchanged.
New startup-only selector auth.local.throttle_storage (auto | memory | db,
default auto = the database whenever database.backend is sqlite or
postgres). The Gateway installs the store once at startup after the
persistence engine; deps._validate_login_throttle_storage warns when a
multi-process deployment keeps memory. Migration 0034_login_throttle
creates the table; config_version 55 -> 56 in config.example.yaml and the
Helm chart. Docs: README, CONFIGURATION.md, AUTH_DESIGN.md, config and
migrations AGENTS.md, CHANGELOG.
* docs(changelog): link the login throttle entry to #6501
* fix(auth): re-check expiry on the row the login-throttle sweep deletes
The bounded cleanup in SqlLoginThrottleStore.record_failure deleted
`WHERE ip IN (SELECT ip ... WHERE <served or stale> LIMIT 200)`. On
PostgreSQL READ COMMITTED the expiry predicate applied only inside the
snapshot-bound subquery: when the DELETE had to wait on a row lock held
by a concurrent check() that was extending the same lock under a raised
lockout_seconds, PostgreSQL re-evaluated only the outer WHERE on the
updated tuple, IP membership still held for the old candidate, and the
now-active lock was deleted; the next attempt from that IP got a clean
record and reached password verification (review of #6501, P2).
Repeat the served/stale predicate on the DELETE target as well
(SqlLoginThrottleStore.sweep_statement), keeping the bounded candidate
subquery, so the re-evaluation sees the updated row's expiry. The other
writes already carry their full compare-and-set predicate on the target
row: _discard and the check() UPDATE both use _matches (ip, fail_count,
locked_at, lock_duration_seconds), so a tuple changed while they waited
no longer matches and the stale decision is skipped. _discard keeps the
equality compare-and-set rather than adding an expiry condition: its
early-release branch deletes a lock whose committed sentence has not
elapsed under the lowered live policy, which an expiry predicate would
wrongly block, and the equality predicate already pins every input of
the decision. reset stays unconditional by design.
Tests: a dialect-compile test asserts the DELETE's own WHERE carries the
expiry predicate for postgresql and sqlite (red on the previous HEAD), and
a PostgreSQL-only test (skipped without TEST_POSTGRES_URI) reproduces the
update-versus-sweep interleaving with two engines, waiting on
pg_stat_activity for the sweep to block before the extender commits.
Docs: sql.py concurrency contract and the migrations guide's 0034 bullet.
* fix(auth): keep an active login lock's clock and fail closed on explicit db
Address the "changes requested" review on #6501.
Lock clock (confirmed defect): record_failure re-stamped locked_at = now
for every failure on an already-locked row, so a sustained attacker's
sentence slid forward with each attempt. Both stores now apply one rule
in one atomic decision: new_count = fail_count + 1; below max_attempts the
IP counts with no lock (a failure after the operator raised the threshold
clears an existing lock); at or over it, a failure during an *active* lock
keeps locked_at and the committed lock_duration unchanged, while a
never-locked or served row starts anew with now / lockout_seconds (so the
failure that reaches a raised threshold starts a fresh lock rather than
keeping a stale locked_at). The SQL upsert expresses it as nested case()
on the row's own values; the memory store mirrors it. The sweeps in both
stores now leave the row being recorded to that decision instead of
deleting a served lock or stale counter first, which split the contract
(SQL would have produced a fresh (1, NULL, NULL) row). base.py states the
rule.
Explicit db downgrade: resolve_login_throttle_store raises SystemExit —
the failure type of the other startup validations in app.gateway.deps —
when throttle_storage='db' resolves to the shared table but no engine is
initialised, so langgraph_runtime refuses to start instead of silently
running every replica on per-process counters; the lazy bare-app path
applies the same rule and installs nothing. 'auto' keeps warning and
falling back. 'db' on a memory database still degrades with a warning
(there is no shared table to insist on).
Non-blocking items: _key() logs a warning when a client key is truncated
to 255 characters (never the key itself); the sql.py docstring covers
wall-clock locked_at across replicas (run NTP; skew shifts evaluation by
that much) and why STALE_COUNTER_SECONDS=24h / SWEEP_BATCH_SIZE=200 were
chosen; a store-level blocking-I/O anchor covers the SQL reset path; the
_validate_login_throttle_storage warning was verified to render
'throttle_storage=memory' for the StrEnum (the gate test now passes the
enum member and asserts that text). reset stays unconditional: a peer
failure that commits after a reset leaves fail_count=1, one genuine
failure after the cleared history, which a new test documents.
Docs: CHANGELOG entry, config.example.yaml, CONFIGURATION.md,
AUTH_DESIGN.md and the throttle_storage field description.
* fix(auth): re-evaluate a login-throttle check whose compare-and-set missed
SqlLoginThrottleStore.check applied its decision (commit the live
duration, or clear a served / released lock) with a compare-and-set
predicate on the snapshot it had read, but ignored the affected-row
count. When a peer changed the row between the read and the write — an
admitted wrong password incrementing fail_count on another replica, or a
successful login clearing the row — the UPDATE or DELETE matched zero
rows, yet check reported the decision as applied: it could return an
extended 60s sentence while the row kept its 1s duration, so the next
check cleared the lock and let password verification through; a live
duration decrease could be dropped the same way; and a row a peer had
just cleared was still reported as locked (#6501 review, P2).
Every compare-and-set write in check now checks result.rowcount. On a
miss the row is re-read and re-evaluated from the top, up to
CHECK_CAS_ATTEMPTS (3) times, applying a fresh compare-and-set decision
on the new snapshot and never clobbering a reset or replacement: a row
that is gone is reported as allowed, a fresh peer lock is reported as
locked, and an incremented count gets the extension or decrease
committed on it. Exhausting the bound fails closed: nothing is written
and the answer is the last snapshot's lock under the live policy, logged
at debug. The read stays in its own short transaction and each write in
a separate one.
The evaluation is now one shared pure function, evaluate_check in
base.py (CheckDecision), used by both stores so the contract cannot
diverge. The memory store keeps a single pass: its compare-and-set
cannot miss because there is no await between the read and the write on
the loop thread, which its docstring now states.
Tests (red first on df26c2acf) use a session proxy that runs the peer
mutation on a second engine right after the snapshot read ends its
transaction: the reviewer's raised-duration interleaving (59.5s reported
and committed, the increment survives, a later check still locked), the
mirrored decrease, early release on the fresh row, a reset landing
between read and write (0.0, no row recreated), and the bound-exhausted
fail-closed case.
* docs(changelog): resolve merge markers left in the Unreleased section
* test(migrations): pin 0035_login_throttle as the head in the 0034 watermark test
Upstream #6477 added test_migration_0034_run_event_seq_watermark.py asserting
0034 is the single head; this branch chains 0035_login_throttle after it, so
the test now pins the current head and walks the chain like its siblings.
* perf(auth): one session per login-throttle check, one transaction per failure
Two non-blocking performance suggestions from the #6501 review.
Single session per check. The router probed the row with store.get()
and then called store.check(), which re-read it: a recorded IP cost two
connection-pool checkouts and two SELECTs per request on the
unauthenticated login path, letting a lockout flood compete with
legitimate traffic for Postgres connections. LoginThrottleStore.check
now takes the policy as an async callable (LoginThrottlePolicy) and
resolves it lazily inside the store: the probe, the policy resolution
and the compare-and-set loop share one session, a clean IP returns 0.0
without ever calling the policy (the config.yaml read is still skipped
for the common case), and a recorded IP resolves it exactly once, also
across compare-and-set retries. The decision is made on a snapshot read
after the resolution: the policy read is a yield point, and a decision
that needs no write has no compare-and-set to catch a row a peer changed
meanwhile — so one session, but still a read after the yield. The
memory store takes the same signature and re-reads its record after the
await for the same reason. The router passes its asyncio.to_thread
policy read as the callable and drops the separate probe; get stays on
the protocol for tests and anchors. No short-TTL verdict cache: it would
mask a successful login's reset for the TTL window, and the single
session already halves the hot path.
Sweep in the upsert's transaction, plus an index. record_failure ran the
housekeeping DELETE in its own transaction and the upsert in a second
one, two commits (two WAL flushes on PostgreSQL) per failed login; the
sweep now rides in the upsert's transaction with one commit, the keep
exclusion already protecting the row being recorded. The new
ix_login_throttle_updated_at (ORM __table_args__ and the unreleased
0035 migration, created after the table and dropped before it, each
guarded separately so a table left by an interrupted attempt still gets
its index) serves the sweep's stale-counter predicate; the served-lock
predicate scans a table the sweep itself keeps small.
Tests (red first on a137fbc47): a clean IP never resolves the policy and
a recorded IP resolves it exactly once (both stores), the SQL store opens
exactly one session per check and commits once per record_failure
(counting wrappers), the migration test asserts the index after upgrade,
after completing a partial upgrade that lacked it, and gone after
downgrade while compare_metadata stays empty; the router-level
"mutated during policy resolution" tests now inject their mutation
through the policy callable.
* docs(changelog): note config_version 57 in the login throttle entry
* fix(channels): fence restart during service shutdown
* test(channels): fence restart during service stop
* test(channels): exercise restart after shutdown channel snapshot
* docs(channels): document service shutdown publication fence
* fix(channels): keep agent guidance within CI hard budget
* fix(channels): fence publication after shutdown completes
* test(channels): cover resume after completed service shutdown
* test(channels): mark direct-start fixtures as active services
* test(channels): mark direct-start fixtures as active services
* style(channels): format shutdown regression test
* fix(channels): drain channel startup completing after service stop
* test(channels): ensure late channel startup is drained after stop
* style(tests): format channel shutdown regression
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scanner): cache decoded text atomically
_cache_text_file keyed the cache on the virtual path and returned the path, and
the caller serves that path as the file's content, so a write that failed
part-way left a truncated entry that every later read handed back as the preview.
Publish through a temporary file plus os.replace.
* test(scanner): pin the cache entry against a failed publish
The helper-level case cannot distinguish the call site, so this drives the real
entry point with a failing publish and asserts the cached entry is untouched.
* docs(sandbox): keep agent guidance within its size limit
* fix(scanner): preserve publish errors during cleanup
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers
DeerMem's derived SQLite FTS5 index for every user was one WAL database pinned
to {storage_path}/.retrieval. With several Gateway Pods sharing one home volume
each Pod opened that file over a network filesystem (unsupported by SQLite),
every Pod start emptied and refilled it under its peers, one Pod's corruption
recovery deleted files the others held open, and a Pod never learned that a
peer had written a user's facts, so memory_search served stale results.
- New backend-private key memory.backend_config.retrieval_index_path: empty
keeps {storage_path}/.retrieval, a relative path resolves against
storage_path, an absolute path is used as-is. The full rebuild and the
corruption recovery act only on that configured index.
- rebuild_index records each agent scope's manifest signature
(mtime_ns, size, revision) before reading its facts; search_facts rebuilds a
scope whose live signature differs because another process wrote the user's
memory. An own commit advances the recorded signatures of that user's scopes
only when they were in sync at the previous revision, so own writes never
rebuild while an interleaved peer write still does.
- Gateway startup warns, in the style of _validate_agent_storage, when a
declared multi-instance deployment keeps the DeerMem index inside
storage_path.
- config.example.yaml documents the key; config_version 55 -> 56, mirrored in
the Helm chart and its README.
- Helm mounts an emptyDir at /var/lib/deerflow/memory-index, points the key at
it, drops the legacy memory.storage_path: memory.json line, and documents the
memory layout next to the other multi-replica prerequisites.
* docs(changelog): link the memory retrieval index entry to #6494
* fix(memory): never advance the index signature on a no-op commit and resync whole scopes
Two review findings on the cross-replica retrieval re-sync:
- save() and apply_changes() inferred the replaced manifest revision as
"returned revision - 1". The commit helper returns the existing manifest
unchanged for a no-op (unchanged-value or empty change set), so after a peer
bumped the manifest an own no-op write advanced this process's stale index
signature to the peer's revision without carrying the peer's delta: a fact the
peer deleted stayed searchable until the manifest changed again. Both paths
now read the real pre-commit revision under the user lock and pass
committed=(pre_revision, signature) only when the commit produced a new
revision; a no-op passes None so the next search still rebuilds the scope.
- The scoped rebuild (signature mismatch and the pre-existing dirty-scope path)
listed facts through list_facts() with its default limit of 100 and then
replaced every row of the scope, so a scope with more than 100 facts lost its
tail on the first re-sync. It now reads the scope's complete fact list.
* fix(memory): fence index signature promotion against a concurrent scope rebuild
Review finding on the cross-replica re-sync: a successful adapter notification
does not prove its delta is still in the index when the signature is promoted.
The SQLite lock covers single adapter calls only, and neither the scope lock
nor DeerMem's warm-up lock spans "notify, then promote", so rebuild_index can
replace a scope's rows in between. After a peer summary-only commit (r+1), A's
search starts a scoped refresh that captures the r+1 signature and reads f1/f2;
A's own deletion of f2 commits r+2 and removes the row; the refresh then
installs its stale f1/f2 and marks the scope synced at r+1; the deletion
resumes and promoted that signature to r+2, so later searches saw the live
signature as equal and kept returning the resurrected f2 until another manifest
change.
Mechanism: a generation fence rather than a lock. rebuild_index bumps a
per-scope generation (scoped rebuild) or a storage-wide generation (full
rebuild) under _cache_lock whenever it publishes or forgets a scope's rows, in
both mark_synced and forget_synced. _dispatch_retrieval_notifications snapshots
the generations under _cache_lock before its first adapter call and, at
promotion time, skips every scope whose generation moved (and everything when
the storage-wide generation moved), leaving the rebuild's own captured
signature in place; that signature predates the commit, so the next search's
comparison rebuilds the scope. A lock would have had to serialize notification
plus promotion with bulk install plus publication while search_facts ->
rebuild_index -> load() already takes the scope, file and cache locks; the
fence adds no lock-order obligations and only costs a redundant rebuild in the
rare race. Every interleaving is covered: a rebuild that completes before the
snapshot is reflected in the captured generation and promotion proceeds on an
index that carries both the rebuild's rows and this write's delta; one that
publishes between snapshot and promotion is skipped; one that publishes after
the promotion records its stale captured signature, which the next search
catches.
Tests reproduce both orders deterministically with threading.Event ordering
(adapter wrapper that holds the bulk install and the remove notification):
test_promotion_is_fenced_against_a_refresh_that_installs_in_between (red before
the fix) and
test_stale_refresh_installing_after_a_promotion_is_caught_by_the_next_search
(regression guard for the mirrored order).
* fix(memory): publish a rebuild's signature under the lock that replaced its rows
Review finding on the cross-replica re-sync: two peer-triggered refreshes of
one scope could replace rows in one order and publish their captured
signatures in the other. On a warmed Pod A, refresh R1 captures r+1 with f1/f2
and waits before installing; a peer deletes f2 at r+2; refresh R2 captures
r+2, installs f1 only and pauses after bulk_rebuild returns, before
mark_synced; R1 installs its stale f1/f2 and publishes r+1; R2 then publishes
r+2 over R1's rows. Later searches saw the live signature as equal and kept
returning the deleted fact. The generation fence only guards own-write
promotion, and the adapter's SQLite lock ends when the install returns.
Mechanism: a per-storage _retrieval_publish_lock held across the row
replacement and the signature publication, for the bulk path (bulk_rebuild +
mark_synced / forget_synced) and the clear+upsert fallback, scoped and full
alike. Overlapping refreshes of a scope therefore serialize their
install+publish pairs: whichever installs last also publishes last, and a stale
captured signature published last differs from the live manifest, so the next
search's compare rebuilds the scope. Fact reading (load() / the glob) stays
outside the lock so a slow read never blocks refreshes of other scopes. The
startup full rebuild holds the lock for its single bulk install; scoped
refreshes wait for it, which is acceptable because before warm-up completes a
search would be rebuilding its scope anyway (DeerMem.warm_retrieval takes its
own _retrieval_lock only after rebuild() returns, so the two never nest).
Lock order: the publish lock is acquired with no storage lock held --
search_facts releases _cache_lock before calling rebuild_index, reload() leaves
its scope/file-lock block before rebuilding, and DeerMem._ensure_retrieval_scopes
holds only its outer RLock, which nothing inside rebuild_index acquires. Inside
the lock only the adapter's own lock and _cache_lock are taken, and both are
leaf locks that never call back into storage, so no cycle is possible.
Test: test_rebuild_publication_stays_ordered_with_row_replacement drives the
reviewer's interleaving with threading.Event ordering (red before the fix). The
pause harness now holds a refresh after its fact read, outside the publish
lock, instead of inside the adapter's rebuild, so the earlier fence tests keep
their semantics without holding the new lock while paused.
* fix(memory): read each manifest once per full rebuild and serve the local index on a compare read error
Two suggestion-level review items on the cross-replica retrieval re-sync.
- The full rebuild cached manifest signatures keyed by (manifest path, agent
name), but _scope_signature reads only the manifest file -- every agent
bucket of a user shares one memory.json -- so the startup scan stat+read+
parsed the same file once per bucket that has facts. The cache is now keyed
by the manifest path alone while synced signatures are still recorded per
(user, agent) scope: users manifest reads instead of users x agents.
- The per-search freshness compare put a shared-volume manifest read on every
search_facts call, and _load_memory_file raises MemoryStorageCorruption on
OSError and malformed JSON, so a transient network-filesystem read error or a
corrupt memory.json failed a search the Pod-local index could still answer.
The compare now catches OSError, ValueError and MemoryStorageCorruption: it
logs a warning naming the scope (traceback at debug), serves the current
local index for that scope and leaves the recorded synced signature
untouched so the next search compares again. It deliberately does not mark
the scope dirty: a scoped rebuild would hit the same failing read, forget the
scope's signature and fail the search path too.
* docs(changelog): resolve merge markers left in the Unreleased section
* fix(memory): forget a scope's index signature when an own delta cannot be proved compatible
Review finding on the cross-replica re-sync: skipping promotion left a newer
rebuild's signature trusted even after this dispatcher had overwritten its rows
with an older notification. On a warmed Pod A, an own PATCH of f2 commits r+1
and pauses after releasing the user lock, before its adapter upsert; Pod B
deletes f2 at r+2; a concurrent search on A detects the peer change, rebuilds
the scope without f2 and publishes r+2; the pending r+1 upsert then reinserts
the deleted fact, and the revision check merely skipped because the synced
revision was r+2 rather than the PATCH's previous r. The signature stayed equal
to the live manifest, so searches returned the deleted fact indefinitely. The
generation fence could not help: it only suppresses promotion, the stale
adapter mutation still happens.
Rule: after an own commit's notifications are applied, the mutated scope's
recorded signature may stay only when the delta is provably compatible with
the published snapshot -- the scope is synced at exactly previous_revision and
no rebuild published rows for it (or storage-wide) since the fence snapshot --
in which case it is promoted as before. In every other case where this
dispatcher mutated the adapter for the scope (synced at another revision,
newer or older; generation moved; not synced at all; or notifications without
a committed revision, i.e. migration and clear paths) the scope's signature is
forgotten so the next search's compare rebuilds it. The user's other scopes
were not mutated by this delta: they are promoted when provable and otherwise
left untouched. The decision runs under _cache_lock together with the
generation reads, so a rebuild publishing concurrently is either observed or
observes the forget; the failure path (dirty + forget) is unchanged.
Orders covered: the reviewer's (publish r+2, then the stale upsert) forgets and
the next search rebuilds without f2; the earlier fence order (publish between
adapter op and promotion) now forgets too, one redundant rebuild; no concurrent
rebuild promotes exactly as before.
Test: test_stale_own_delta_replayed_over_a_newer_snapshot_forgets_the_scope
holds the PATCH's adapter upsert after the user lock is released (new
pause_next_upsert hook on the test adapter), Event-ordered, red before the fix.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(browserless): send the web_fetch token as a query parameter
fetch_html_with_status put the configured token in the /content JSON body.
Browserless reads the token only from the token query parameter or the
Authorization header, and checks it before reading the body, so with
BROWSERLESS_TOKEN set every web_fetch failed with HTTP 401; against an
instance started without TOKEN the unknown body key failed schema
validation with HTTP 400. capture_screenshot already sent ?token=, so
web_capture worked with the same token.
Both endpoints now take the token from BrowserlessClient._auth_params().
The test that pinned the body token is rewritten, and a stand-in for the
Browserless request gate (query/header auth, no extra body keys) drives
both endpoints through the real httpx request path.
* docs(changelog): reference #6484
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(events): preserve thread seq watermark across run deletion
* fix(events): clean up watermarks for empty threads
* perf(events): avoid duplicate fence on run deletion
delete_by_run already reaches _max_seq_for_thread, which acquires the transaction-scoped thread mutation fence. Keep the count and delete under that same transaction lock without taking the Postgres advisory lock twice, and pin the single acquisition in the existing test.
---------
Co-authored-by: yu1128-L <296164533+yu1128-L@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): fail fast when DEER_FLOW_HOME is not writable
`make docker-start` bind-mounts the host `backend/` directory into the gateway
container, whose process runs as root, so any state it creates under
`backend/.deer-flow` ends up owned by root on the host. A later `make up` runs
`scripts/deploy.sh` as the invoking user and only fails deep into the run with a
bare "Permission denied" while persisting generated secrets (deploy.sh:234).
Check writability right after `mkdir -p "$DEER_FLOW_HOME"` and, when it fails,
print the exact `chown` recovery command instead of letting the run get halfway
through. `make down` is excluded so stopping the stack is never blocked.
Fixes#6462
* test(deploy): cover runtime home writability preflight
* fix(deploy): diagnose persisted secret permissions and preserve teardown
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): keep a slash skill activation when the model call is retried
SkillActivationMiddleware records the activated slash message in the run
context before it calls the model. LLMErrorHandlingMiddleware sits outside
it and retries a failed (429/503/timeout) or empty call by running this wrap
again on the same request; the retry found the run key already set and went
out without the SKILL.md body, while the slash source it had published kept
the skill's tool restrictions in force.
Record the activation itself next to the run key and replay it while nothing
has answered the slash message yet: a retry sees no AI message after the
target, a tool-loop follow-up call does. The replay carries the same reminder
and usage stamp without re-reading the skill or repeating the audit event and
usage record, which now sit with the other once-per-activation side effects.
Two existing tests sent the same unanswered request twice to model a
follow-up call; that is the retry shape, so their second call now includes
the first response, as the real graph does.
* docs(changelog): reference #6506
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): share skill cache invalidation across Gateway replicas
SkillStorage.load_skills() rescans disk and re-reads extensions_config.json
on every call, but the prompt layer in deerflow/agents/lead_agent/prompt.py
caches the enabled-skills list, the per-(app_config, user_id) entries and
the rendered <skill_system> section per process, and every skill mutation
in routers/skills.py only refreshed the process that handled it. With
several uvicorn workers or several Pods sharing one home volume, a skill
installed, edited, deleted, rolled back or toggled through replica A kept
being offered as the old skill set by replica B until B restarted, and
POST /api/skills/reload had to be sent to every Pod separately.
Reuse the MCP cache-reset mechanism. A new generic helper,
deerflow.config.shared_reset_marker, owns a hidden JSON marker beside the
resolved extensions config (.<config name>.<suffix>.json): publish()
replaces it atomically with a random generation under
extensions_config_write_lock + extensions_config_file_lock, chaining
previous_generation and an optional user_id; SharedResetMarkerTracker
polls it at most once per second (monotonic clock, never blocking behind a
concurrent poller), adopts the current state silently on its first poll,
and classifies a change as user-scoped only when exactly one publication
carrying that user happened since the last poll. deerflow/mcp/cache.py now
builds its .<config>.mcp-cache-reset.json marker through the same helper
with no behavior change.
prompt.py publishes .<config>.skills-cache-reset.json via
publish_skills_cache_reset(user_id=...) and checks the marker on every
cache lookup (global cache, per-config cache, prompt-section LRU, user
scoped entries), calling _invalidate_enabled_skills_cache() or
invalidate_user_skill_cache(). The publishing process remembers its own
generation so it does not retire the caches it just rebuilt. Every
mutation tail in routers/skills.py (install/upload, edit, delete,
rollback, enable/disable, reload) refreshes locally and then publishes the
marker off the event loop; /skills/reload answers scope=shared_config,
or scope=process when no extensions config path resolves, and a marker
publication failure is a 500 so a 200 never means only one process was
refreshed.
Docs: README, backend/docs/API.md, skills/config/mcp/gateway AGENTS.md,
and frontend skills.mdx (en/zh; also fixes the reference to a
non-existent skills/loader.py) describe the shared invalidation and drop
the per-Pod guidance. CHANGELOG entry under Unreleased / Fixed.
* docs(changelog): link the skills cache reset entry to #6495
* fix(skills): drain the reload tail and keep marker failures out of 404
Two review follow-ups on the shared skills cache reset (#6495).
POST /skills/reload awaited the local refresh and the marker publication
unshielded. CancelledError is a BaseException, so a caller that
disconnected between the two slipped past the `except Exception -> 500`
mapping, surfaced no error, and left every peer replica serving the old
skill set -- the one outcome the operator hook for external mount writes
exists to prevent. The refresh and the publication now run as one
`await_drained(_drain_skill_mutation("reload", ...))` tail, the same
discipline the install/edit/delete/rollback/toggle tails already use; the
exception -> 500 mapping and the scope response are unchanged.
Publication errors could surface as 404. resolve_shared_config_path()
only swallows the resolution-time FileNotFoundError; a config directory
that vanishes between resolution and the lock file / atomic replace still
raised FileNotFoundError from the marker write, and the install, edit,
delete and rollback handlers mapped that to HTTP 404 with the raw OS
error text for a skill that had in fact been applied. publish_skills_cache_reset()
now re-raises OSError (and a misconfigured project root's ValueError) as
SkillCacheResetPublishError, a RuntimeError that no handler maps to
404/400, so every handler reports it through its generic 500 branch; the
message names only the exception type, never a server path. The install
handler gains an explicit clause that pins that contract beside its 404
mapping.
Tests: cancelling reload after the refresh still publishes the marker;
a FileNotFoundError from the marker write yields 500 (not 404) for
install, edit, delete and rollback, 500 for the public toggle and the
generic reload error; the wrapper type is not an OSError.
* fix(middleware): guard OverflowError when parsing non-finite Retry-After hints
* fix(middleware): guard OverflowError when parsing non-finite Retry-After hints
* fix(retry): bound provider retry hints across model paths
Share a 24-hour ceiling for middleware numeric/date hints and Claude's
integer-second hints. Non-finite or oversized values use local backoff;
valid hints above the jitter cap remain unchanged and negatives clamp to zero.
Parameterize unusable hints at both middleware test levels, including nan.
Cover numeric/date boundaries and sync/async recovery through both retry paths.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): skip task servers whose discovery failed instead of dropping all tools
Per-server discovery replaced a failed or timed-out server with an empty
tool list. For a server with task_toolsets, _configure_task_tools_for_server
then reported submit/status/cancel as missing and raised
McpTaskConfigurationError, which get_mcp_tools re-raises, so the tools of
every healthy server were discarded and the cache was never published:
each agent build rediscovered every server.
load_server_tools now returns None for a failed discovery, distinct from a
server that answered with zero tools, and the wrapping loop skips such
servers before task-toolset validation. A server that answers without its
configured raw tools still raises as a configuration error.
* docs(changelog): reference #6481
* docs(mcp): name the cache-reset endpoint as a task-server recovery path
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): fall back when a token response carries an unusable expires_in
OAuthTokenManager._fetch_token() converted the token endpoint's
expires_in to a timedelta while guarding only TypeError/ValueError. Two
server-controlled inputs raised OverflowError, which escaped the parser and
failed the whole token fetch, so the MCP server never received its
Authorization header:
* int(float('inf')) raises OverflowError. json.loads accepts Infinity, so a
non-finite expires_in reached it uncaught (NaN happened to be caught).
* int(1e30) succeeds, but timedelta is capped at 999_999_999 days and rejects
the magnitude with OverflowError. A large JSON integer literal behaves the
same way.
Catch OverflowError alongside TypeError/ValueError and treat a lifetime beyond
timedelta's own range as unusable, so both fall back to the documented
one-hour default like every other malformed value.
Fixes#6472
* fix(mcp): handle both token expiry overflow stages
---------
Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(migration): read thread owners from the configured database
migrate_user_isolation.py looked for thread owners in
{base_dir}/deer-flow.db, a file DeerFlow never creates. The database is
{sqlite_dir}/deerflow.db or PostgreSQL, so the owner map was always empty
and every legacy thread was moved to users/default/.
Read threads_meta through the configured database's sync SQLAlchemy URL
(SQLite or PostgreSQL, honoring postgres_schema). When legacy threads
exist but owners cannot be read (no database file, a memory backend, a
missing driver, or a failed query), stop before migrating anything;
--allow-missing-thread-owners keeps the old assign-to-default behavior
for installs that never recorded owners.
* docs(changelog): reference #6450
* fix(migration): document recovery and report unusable database URLs
Installs that already ran the old script have every legacy thread under
users/default/threads/, and re-running finds no threads/ to migrate. The
provisioner README and AUTH_DESIGN now give the recovery: move the owned
threads back to threads/ and re-run; an existing owner directory is never
overwritten and the old copy goes to migration-conflicts/. A test pins
that re-run behavior.
create_engine errors other than ImportError (e.g. ArgumentError from an
empty postgres_url) now stop with the same "Nothing was migrated." message
instead of a traceback.
The memory AGENTS.md note said threads go to --user-id; they go to their
threads_meta owner.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(models): validate Claude extended thinking budgets
* fix(models): preserve Claude thinking budget semantics
* fix(models): isolate Claude auto thinking budgets
* fix(models): preserve interleaved Claude thinking budgets
Consolidate the additional thinking-budget boundary and type coverage from PR #6482 into #6060. Keep copied automatic budgets and preserve valid manual interleaved budgets with supported models, tools, and the effective beta header. Document the validation scope and cover native payload construction and SDK header precedence.
* docs(models): avoid upstream Claude setup overlap
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The extension notify loop told a spent shared budget apart from a
TimeoutError a contributor raised on its own by re-reading loop.time()
after the await and comparing it with the deadline. The event loop runs a
scheduled timeout up to one clock_resolution early (BaseEventLoop._run_once
compares each handle's _when against now + clock_resolution), so on a coarse
clock -- Windows' ~15.6 ms against budgets as small as 20 ms -- that read can
still land strictly before the deadline. Budget exhaustion was then logged as
a phantom hook failure (ERROR plus an asyncio-internal traceback), and the
same lag let a later contributor run on a budget that had already run out.
Classify from asyncio.timeout's own expiry (expired()) instead of a second
clock read, and latch budget_spent once the budget is gone so the skip of
later contributors is deterministic too.
Fixes#6453
Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
* test(auth): make user-list ordering independent of clock resolution
* docs(test): clarify deterministic user-list ordering
Explain the lexical tie-break over stored canonical UUID strings and give
user-repository ordering its own guidance section, separate from router fixtures.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skillscan): classify env command launchers
* fix(skillscan): bound and correct env launcher parsing
Resolve env split strings iteratively with bounded work, preserving critical
findings on limit exhaustion. Tokenize each simple command in place and keep
quoted argv words distinct from adjacent redirection operators.
Honor env assignments after option markers, GNU split-string escapes, absolute
env paths, command/exec/sudo wrappers, and the non-exiting --debug option. Scope
stdin overrides to the pipeline consumer. Add 60 parser and scan regressions,
including recursion, many-command, and deep-group deadline cases.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): keep output and exit code when an e2b command fails
* docs(changelog): link #6441
* fix(sandbox): keep list_dir output when an e2b listing exits nonzero
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The no-tool native stream applied _decode_escaped_newlines_outside_fences
to each streamed chunk independently, but a fenced code block almost never
opens and closes within one token-sized chunk, so the per-chunk fence regex
never saw a complete ``` pair and decoded every literal \n inside the fence.
Code like print("a\nb") reached the streamed answer and the persisted
message content with a real newline, while the non-streaming path and the
tool-mode simulated stream preserved it.
Decode the native stream with a fence-stateful decoder instead: it carries
whether the stream is inside a fence across chunks and holds back a
trailing partial ```/`\n` sequence, so the joined stream equals the one-shot
decode of the same reply. A fence left unclosed by a truncated reply keeps
its escapes, matching how an unterminated fence still renders as code.
Document native no-tool streaming in the MindIE decode contract.
* fix(goal): fall back to sync methods on sync-only checkpointers
The goal helpers called any async checkpointer method that existed. Every
BaseCheckpointSaver defines the async methods, and the sync SqliteSaver and
PostgresSaver used by the TUI and embedded DeerFlowClient raise
NotImplementedError from them, so get_goal/set_goal/clear_goal always failed
there and the TUI printed "Could not set goal.". Fall back to the sync method
only on NotImplementedError; other async errors still propagate and sync
writes stay drained across cancellation.
* docs(changelog): reference #6448
* fix(goal): log when a checkpointer call falls back to the sync method
The fallback was silent, so a sync-only saver wired into the Gateway would
work through the thread-offloaded path with no trace. Emit a debug record
naming the saver and method; debug rather than warning because the TUI and
embedded client take this path on every call.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): keep in-flight staging files out of the startup sweep
`cleanup_stale_upload_staging_files` removed every `.upload-*.part` it
found at Gateway startup. That assumed a single Gateway: nothing can be
uploading while it boots, so every staging file is a crash leftover.
With several replicas sharing one home volume (Helm RWX), a replica
starting during a rolling update deleted the staging file of an upload
another replica was still writing; that upload then failed at its
`os.link` commit and lost its bytes.
The sweep now skips staging files younger than `UPLOAD_STAGING_MIN_AGE`
(24 hours, the same guard as the project-document `.staging` orphan
sweep in `projects/trash.py`). Chunk writes refresh the mtime, so an
upload in flight stays younger than the guard; an unreadable mtime keeps
the file. The startup call site is unchanged and uses the default.
Tests: the existing sweep tests backdate their orphans; new cases pin
young-kept / old-removed, the configurable `min_age`, unreadable and
vanished mtimes, and a router test that runs the sweep while an upload
is blocked mid-write and asserts the upload still commits.
* docs(changelog): link #6445
* fix(uploads): reclaim published staging aliases at any age
Review follow-up (#6445): a Gateway that dies after the commit's
`os.link` but before the staged-name removal leaves the final file and
its `.upload-*.part` alias on one inode. With only the age guard, the
next startup kept that alias for 24 hours, and because the sweep runs
only at startup, `copy_upload_file_no_symlink` (embedded
`DeerFlowClient.upload_files`) kept rejecting the destination for its
multiple links.
The sweep now stats each staging name with `os.lstat` (a `DirEntry`
stat reports `st_nlink == 0` on Windows) and removes a name with
`st_nlink > 1` at any age: a staged part only ever gains its second
link through its own commit, so such a name is a published alias, never
an upload in flight. Lone parts keep the 24 hour mtime guard. The
destination hardlink check is unchanged.
Tests cover the alias next to a young lone part, the replacement upload
succeeding afterwards, and the crash reproduced through the real upload
path with the staged-name removal patched out.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): probe the Redis stream bridge and provisioner in /health/ready
GET /health/ready only probed the ORM engine and the LangGraph checkpointer.
The Redis stream bridge client connects lazily and nothing ever pinged it, so a
Gateway whose Redis was down started, answered 200 ready to the Kubernetes
readinessProbe and Compose healthcheck, and kept receiving traffic while every
run's first publish failed. Since #6328 the Redis bridge is mandatory for
multi-instance deployments, so such a replica is not serviceable at all.
- StreamBridge gains `async ping() -> bool`; the base returns True (no external
backend), RedisStreamBridge round-trips PING and answers False on RedisError.
- /health/ready adds a `stream_bridge` probe (not_configured for the memory
bridge; unreachable flips the endpoint to 503; a missing bridge fails closed)
and a report-only `provisioner` probe of `{sandbox.provisioner_url}/health`
that never changes the status code, since every replica shares one
provisioner. The provisioner URL is snapshotted on app.state at startup.
- A probe that overruns the endpoint deadline is now the only one marked
unreachable; finished verdicts are kept.
- The public probe gate is one lock per probe kind, so one request's probes
still run concurrently while a thundering herd opens at most one probe
connection per kind per process.
- Tests: bridge ping cases, stream-bridge/provisioner probe cases, verdict and
deadline cases, and the first route-level /health/ready tests via TestClient.
- Docs: gateway AGENTS.md, CONFIGURATION.md multi-instance notes, Helm README
and probe comment, Compose healthcheck comment.
* docs(changelog): note the readiness probe fix (#6447)
* fix(gateway): apply the sandbox proxy policy to the provisioner probe
Review follow-up for #6447.
- The provisioner readiness probe built its httpx client with the default
trust_env=True. httpx 0.28 treats CIDR entries in NO_PROXY as host patterns
rather than subnet bypasses, so with HTTP_PROXY set and NO_PROXY=10.0.0.0/8
a private provisioner that the requests-based sandbox calls reach directly
would have been probed through the proxy and reported unreachable on every
check. The probe now reuses `sandbox_http_trust_env`: loopback, private,
link-local and cluster-local addresses bypass the proxy, external hosts keep
the environment's proxy settings.
- Pin the liveness/readiness separation with a route test: `GET /health`
stays 200 while the stream bridge and database are unreachable and
`/health/ready` is 503, so a Redis blip can never become a restart loop.
* feat(wizard): add Opper as an LLM provider
Opper (https://opper.ai) is an EU-hosted AI gateway: 700+ models from
50+ providers behind one OpenAI-compatible API and one key. This adds it
to the setup wizard's LLM_PROVIDERS next to OpenRouter, with the same
shape: langchain_openai:ChatOpenAI pointed at
https://api.opper.ai/v3/compat, env var OPPER_API_KEY.
Models use Opper's bare model ids: claude-sonnet-4-6 (default), gpt-5.5,
gemini-3.8-flash and gpt-5.4-mini. Because the GPT ids are bare,
langchain-openai drops the 0.7 temperature for them the same way it does
for direct OpenAI gpt-5 models.
Checked: python -m py_compile scripts/wizard/providers.py, and each
model id against Opper's public model list
(GET https://api.opper.ai/v3/models). Not run locally: make lint and
make test.
Disclosure: Opper is my startup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs: add Opper to the config example and READMEs
Add an Opper model example next to the other OpenAI-compatible
gateways, as requested in review:
- config.example.yaml: commented opper-claude-sonnet-4-6 block after
the other gateway examples (ChatOpenAI + base_url, OPPER_API_KEY)
- README.md and the translated READMEs: the same entry in the manual
model configuration examples, after OpenRouter
- .env.example and backend/docs/CONFIGURATION.md: OPPER_API_KEY, in the
shape of the Novita lines
claude-sonnet-4-6 is a pooled id in Opper's public catalogue
(GET https://api.opper.ai/v3/models). config.example.yaml still parses
with yaml.safe_load; config_version is unchanged because the schema is.
Disclosure: Opper is my startup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- secret-cloud-token: add the github_pat_ and AIza families, matching the
canonical PII detector.
- _is_shell_path: treat .zsh by suffix, so a missing shebang or a rename can
no longer bypass every shell rule.
- Sync the same two patterns into _EGRESS_SECRET_PATTERNS so the IM egress
scrubber stays in lockstep with skillscan.
- Regression tests for all of the above.
Co-authored-by: yu1128-L <296164533+yu1128-L@users.noreply.github.com>
* docs: run configuration and migration commands in the backend environment
* docs: preserve runtime paths and extras in backend commands
---------
Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
test_json_output_uses_concise_review_record_schema asserts the JSON record
against str(source_file). The detector builds location.path through
relative_to_repo, which is as_posix() on both of its branches, so on POSIX the
two strings are equal and the assertion holds. On Windows str(path) yields
backslashes, so the test fails against output that is behaving as documented.
One word: str(source_file) becomes source_file.as_posix(). On POSIX the two are
the same string, so this is a no-op for CI; on Windows it makes the assertion
match what the detector emits. Every other path field in the repository is
written with as_posix() as well.
Before: 1 failed, 50 passed. After: 51 passed, on Windows with Python 3.12.
The three sibling detector suites still pass (35 passed), and ruff check and
ruff format --check pass on the file.
retry_item unconditionally flipped batch.status to queued, and claim_items
only schedules queued or running batches. Retrying one failed item in a
paused batch therefore lifted the pause barrier and let the scheduler
take every unrelated pending item without an explicit resume.
Leave the batch row untouched when it is paused; the retried item is
still reset to pending and waits for resume_batch like its siblings.
Non paused batches keep the existing requeue behavior.
Fixes#6411
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
