* feat(opensandbox): add opt-in secure_access sandbox option
Forward a new `sandbox.secure_access` option (default false) to
SandboxSync.create so OpenSandbox can require a per-sandbox endpoint
token at the ingress gateway and server proxy. The server returns the
token as an endpoint header and SDK 0.1.15 already attaches endpoint
headers to execd and file requests, so DeerFlow needs no token handling.
Non-boolean values are rejected at provider startup so a quoted "false"
cannot silently enable it. Documented in config.example.yaml,
CONFIGURATION.md, the SandboxConfig docstring and the provider README.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(opensandbox): validate server proxy boolean
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Totoro <msy626836554@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(browserless): reject timeouts httpx cannot use
`_coerce_timeout` forwarded any int, float or numeric string to
`httpx.AsyncClient(timeout=...)` unchanged, so a configured `timeout_s: 0`
or a negative value failed every fetch and capture with `ConnectTimeout`
against a healthy backend, `nan` raised `ValueError: deadline must not be
NaN`, and `inf` left the request without a deadline so it never completed.
Each of those silently disabled the provider.
Fall back to the documented 30s default with a warning when the number is
not a usable deadline. This matches `crawl4ai._coerce_timeout` after #7481;
`jina_ai._coerce_timeout` still forwards non-positive values.
* fix(browserless): address timeout review and format tests
Explain how HTTPX handles NaN and positive infinity. Document that
Browserless accepts positive finite timeouts without an upper limit,
and pin that behavior with a 1e308 test case.
Crawl4AI and Jina timeout validation are tracked in #7481 and #7823.
Remove the Browserless docstring's claim that Crawl4AI shares its
coercion behavior. Format the test file for the backend lint check.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs: turn README into an index and move reference detail to docs/guide
The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md.
* docs: send user docs to topic guides and budget the root README
The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003).
* chore: remove unreferenced screenshots and finished plan notes
pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them.
* docs: route always-loaded AGENTS.md files to scoped guides
* docs(agents): cut scoped guides to scope, verify, traps, decisions
Each AGENTS.md keeps four kinds of content: the scope of the directory, the
verification commands, traps a local test run does not show, and decisions the
code cannot state. Values, field lists, function inventories, test names, issue
numbers and history are deleted, because the code, the tests and git state them.
Search and fetch providers share one community guide. Four guides with no
remaining content are removed. The size check now allows 6 KiB per file (hard
limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB).
* style: format frontend guide table and guidance test imports
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): include nginx logs in dev_multi_instance.sh logs all
`logs all` selected only the two Gateway logs, the same as running
`logs` with no argument, so nginx's error and access logs never showed
up although the help lists all next to nginx. `all` now adds the nginx
logs, and the nginx selection also covers nginx.log, where the start
command's own output goes and which no option printed before.
* docs(changelog): reference #7723 for the dev_multi_instance logs fix
* test(scripts): pin dev_multi_instance logs when nginx never started
Cover the layout start_nginx leaves behind when it bails on a missing
binary or a busy port: only gateway-*.log exists, so `logs all` prints
the two Gateway files and `logs nginx` prints nothing, both exiting 0.
Declare required thinking, native reasoning effort and image support for both
K3.1 Pro IDs. Normalize fixed parameters at the final Moonshot request boundary
and cover real SDK serialization with offline regression tests.
Fixes#7425
* fix(runtime): present outputs before a loop wrap-up and explain step-limit stops
Loop-detection warnings now ask the agent to present the files it already
created in /mnt/user-data/outputs before its final answer, when
present_files is available.
A run that exhausts recursion_limit now ends with stop_reason
recursion_capped and a plain message that lists outputs it created but
did not present. An active goal stands down without an evaluator call.
The /wait error envelope keeps the GraphRecursionError name, and IM
channels reply with the step-limit message instead of a generic failure.
* docs(changelog): note run-end delivery and step-limit stops (#7588)
* test(runtime): stop pinning LangGraph's step count in the step-limit test
The produced paths and the message already follow the observed model
calls, so the exact count only pinned LangGraph's per-step accounting.
Also note on the step-limit goal branch that its caller passes no
evaluator factory, so it has to stay ahead of the evaluator section.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(streaming): make Redis stream END publication idempotent
* fix(streaming): align Redis test fake with Lua END checks
Remove the unreachable owned missing-stream branch, mirror tail-only data checks in both fake script paths, and pin legacy END and owned missing-buffer behavior against real Redis.
* fix(streaming): gate legacy END scans after writer rollout
* fix(helm): synchronize embedded config version with schema
* test(runtime): use real stream bridge config in close coverage
Keep new configuration defaults in lifecycle fixtures and surface owner-task failures instead of masking them as event timeouts. Drain every helper task and cover context entry/close failures.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): honor the configured recursion limit in IM runs
Every IM channel run asked the Gateway for recursion_limit 100, because
the channel manager seeded each run config with it. The Gateway honors a
request value, so the top-level recursion_limit never reached IM runs,
and a budget of 100 could stop a run after a handful of tool-calling
turns (six in one measurement with the default lead-agent middleware;
the count changes with the middleware chain).
IM runs now send no recursion_limit of their own and get the Gateway's
configured default, clamped by max_recursion_limit, like Gateway API
runs. A session config value still takes precedence, and GitHub channel
runs keep their floor of 250. The README and config.example.yaml IM
examples no longer pin session.config.recursion_limit to 100.
* docs(changelog): reference #7521 for the IM recursion-limit fix
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): normalize mount container_path trailing slashes at config load
A container_path declared with a trailing slash (e.g. /mnt/reports/) was
accepted by VolumeMountConfig and normalized by LocalSandboxProvider when
building path mappings, but the sandbox tool path gates compare the raw
configured value, so every ls/read/write/glob/grep/bash access to the
mount failed with a confusing PermissionError. Normalize trailing
slashes in the config model so both layers agree, and reject the
container root '/' which the gates cannot express.
* fix(sandbox): normalize skills.container_path and pin empty-value rejection
Review follow-up on #7393:
- Share the container-path normalizer with SkillsConfig, whose prefix
feeds the same as-is comparison in _is_skills_path and had the same
trailing-slash dead-path failure mode.
- Pin the empty-string rejection: on main an empty container_path made
the gates match every absolute path, applying one mount's read_only
flag to all paths.
- Cover both config fields in CONFIGURATION.md.
---------
Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com>
The two-Gateway harness checklist in CONFIGURATION.md said a crashed
leader hands its providers over within lease_seconds + grace_seconds; a
peer only tries to acquire on its fixed lease_seconds // 3 tick, so the
worst case is one renewal interval more (50 s with the defaults), as the
leader-lease section of the same file already says. The grace_seconds
field description had the same shorter bound.
The channels.leader_lease comment in config.example.yaml still said web
UI credential changes reach the holder only when it next starts the
channel; since #6988 the holder applies them within one renewal interval.
The harness `check` footer still listed the IM channel gates as waiting
for leader election, which landed in #6843; it now points at the manual
channel item in CONFIGURATION.md, which also gains the runtime-config step.
* fix(stream): tolerate ~~~-fenced code and \-prefixed closer when decoding escaped newlines
_decode_escaped_newlines_outside_fences only recognised ``` fences, so replies
that used ~~~ fences (e.g. code blocks rendered by the model with
~~~python ...) kept literal backslash-n sequences escaped instead of decoding
the ones outside fences, and a fence whose closer was split as backslash+tilde
across chunks was misread mid-stream. Fixes#7262.
- detect the opening fence with (?:```|~~~) and remember which delimiter the
fence was opened with, so closing requires the same delimiter
- while inside a fence, hold back a closer that begins with a backslash so a
fence ending in backslash+tilde mid-stream is not misread; the held tail is
revisited on the next push and normalised on flush
- keep backtick hold semantics unchanged
* fix(mindie): preserve unclosed fences during newline decoding
Match one-shot decoding to streaming by choosing the earliest backtick or tilde opener and preserving the remainder when its matching closer is absent. Correct the string-return docstring and cover mixed delimiters across every two-chunk split and character-by-character streaming.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Exa web_search and web_fetch accept a per-tool base_url, falling back to
EXA_BASE_URL and then the SDK default. Overrides are validated like the
Serper base URL. web_fetch screens a configured backend with the shared
delegated-backend guard.
Co-authored-by: lachy3433 <lachlanallen.co@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mcp): keep rotating OAuth refresh tokens across restarts and Gateway instances
MCP servers on the OAuth refresh_token grant lost their rotated refresh
token whenever a new process or manager started (restart, tool-cache
reset for non-task servers, every personal-connection run, a second
Gateway instance) and re-seeded with the consumed configured token,
getting invalid_grant for good.
- New table mcp_oauth_tokens (migration 0047_mcp_oauth_tokens), one row
per refresh_token connection keyed (scope, owner_user_id, server_key):
('deployment', '', name) or ('personal', owner, personal runtime
name). Access and refresh tokens are DEER_FLOW_CREDENTIALS_KEY
ciphertext; a config fingerprint (token_url, client_id, grant, scope,
audience, seed) makes a changed seed win.
- OAuthTokenManager re-reads the row under its process lock, adopts a
peer's valid token, and otherwise claims the refresh with a lease plus
a version compare-and-set (database clock), fetches outside any
transaction, writes under the lease and releases it on failure.
- The Gateway installs the store (sync engine via to_thread) around the
LangGraph runtime for sqlite/postgres; personal managers pass their
owner. client_credentials stays in process; an unloadable key falls
back to process memory; database errors fail the refresh.
- credentials_key_consumers lists refresh_token MCP OAuth servers, so a
declared multi-instance deployment needs the shared key.
Docs: MCP_SERVER.md, CONFIGURATION.md, Helm README, README, mcp and
migrations AGENTS.md, CHANGELOG (en/zh).
* docs(changelog): link the MCP OAuth token store entry to #7207
* fix(mcp): prune removed connections' OAuth token rows and ask owners to re-authorize on invalid_grant
Review follow-up (willem-bd):
- Both MCP config writers drop mcp_oauth_tokens rows of connections a
save removes (deployment: _commit_mcp_config_write; personal:
_mutate, keeping each connection's enabled/disabled runtime-name
variants so a toggle keeps its chain). Best effort; no startup sweep,
since a missing config file would look like every server removed.
- invalid_grant on a refresh_token request raises
McpOAuthReauthorizationRequired: the tool error names the server and
asks for a new refresh token, plus the shared-key hint when this
instance could not decrypt the stored chain.
- Annotate the gateway helpers' config as AppConfig.
* fix(mcp): renew the OAuth refresh lease, keep unstored rotations, and key access tokens by request parameters
Review follow-up (willem-bd, second round):
- HTTPX timeouts bound each I/O wait, not a whole response: the refresh
holder now renews its lease every third of REFRESH_LEASE_SECONDS while
the token request runs and abandons it after REFRESH_MAX_SECONDS (120 s);
waiters outlast a live holder. Regression uses a real loopback endpoint
that streams its body for three lease lengths.
- A refresh whose tokens could not be stored keeps them pending
(unsaved_base_version): they are written back before an older stored
token is adopted or refreshed with, retried every few seconds while the
access token is still valid, and dropped once a peer stored a newer
chain. A failed save releases the lease.
- New token_request_fingerprint (extra_token_params and response fields):
a stored access token is reused only under the same request parameters;
otherwise the stored refresh-token chain is refreshed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(deploy): select production Compose env files
Use an operator-selected dotenv file consistently for interpolation, container environment, secret probes and build extras. Preserve default selection and shutdown after a profile is removed.
Co-Authored-By: Codex <noreply@openai.com>
* fix(deploy): address production env selection review
Clarify Docker launcher entry points and render real Compose through the launcher to verify both service environments.
Co-Authored-By: Codex <noreply@openai.com>
* test(deploy): cover unreadable env selectors
Reject unreadable explicit selectors before Docker is invoked, using POSIX permission checks and restoring the fixture mode in finally.
Co-Authored-By: Codex <noreply@openai.com>
* test(deploy): scope dotenv assertions to selected secrets
---------
Co-authored-by: Codex <noreply@openai.com>
* fix(memory): persist the DeerMem extraction watermark across restarts and replicas
The updater's per-thread extraction watermark lived only in an in-process
LRU, so a restart or a second Gateway replica on the same storage root fed
already-extracted turns to the memory LLM again.
FileMemoryStorage now keeps it in users/{user}/agents/{agent}/.metadata/
watermarks.json, keyed by thread id, written atomically under the per-user
lock without touching memory.json and capped by watermark_max_keys. Each
update feeds after the furthest of the LRU entry and the persisted one, and
writes through after an extraction or a pre-screen skip; bypass_watermark is
unchanged and sidecar IO failures fall back to the LRU. Memory clears remove
the file with the other .metadata sidecars. Content-based message identities
are stored as SHA-256 digests so the sidecar holds no conversation text.
* docs(changelog): link the memory watermark entry to #7204 and #7205
* fix(memory): never replace an unreadable watermark sidecar
set_extraction_watermark rewrites the whole watermarks.json, and the shared
sidecar reader turns an OSError into an empty mapping, so a failed read could
replace every other thread's watermark with the current one (or delete the
file). The watermark sidecar now has its own reader: a missing file reads as
empty, malformed content reads as empty (nothing to keep; the next write
repairs it), and any other OSError propagates before anything is written. The
updater already logs that and keeps the extraction successful.
* fix(memory): keep the persisted watermark when the last message has no identity
_message_identity returns None for a last message with neither an id nor
text. The updater wrote that None through, which deleted the thread's entry
from watermarks.json, losing a watermark a peer may have advanced and forcing
a full feed on every replica. The write-through now skips None: the persisted
entry still marks an extracted prefix and the furthest-of-both read keeps
using it.
* refactor(skills): keep allowed-tools as compatibility metadata
* docs(skills): align current guides with agent tool authority
* test(skills): simplify authority regressions and sync Korean docs
* fix(channels): run each IM message once across instances, restarts and late redeliveries
Inbound IM messages were deduplicated only by the 10-minute inbound dedupe
store (#7103). Discord direct messages had no key at all, a Discord thread
reply got an instance-dependent key, a Discord thread was created before
dedupe ran, a redelivery past the window started a second run, and two
instances racing on a conversation's first messages bound it to two
threads.
- Discord messages are keyed (discord, guild_id or "dm", posting channel,
message_id); the posting channel is the same on every instance.
- MessageBus.claim_inbound lets an adapter record the manager's dedupe key
before a side effect; Discord claims it on the Gateway loop before
_create_thread, marks the published message dedupe_claimed, and gives the
claim back when the message is never committed.
- runs.wait / runs.stream / runs.create send an Idempotency-Key derived from
the dedupe key, and keyed human message ids are uuid5 of it (a uuid4 per
attempt made every keyed retry a 409). A reused run's status-only /wait or
stream gap is resolved to this message's own turn via threads.get_state;
409s about the message's own keyed run keep the key instead of reading as
a busy thread; an ended reused GitHub run gets no follow-up watcher.
- A conversation's first thread binding is claimed insert-if-absent
(claim_thread_id on channel_conversations and channel_thread_bindings);
the loser adopts the winner and deletes the empty thread it minted.
/new and /agent use still rebind.
Docs: IM_CHANNEL_CONNECTIONS.md, CONFIGURATION.md, channels AGENTS.md,
CHANGELOG (en/zh).
* fix(channels): give back Discord dedupe claims for messages never processed
Review round 1 on #7104. A Discord claim records the manager's dedupe key
before the message is queued, so every path that drops the message unrun
must release it, or a peer refuses the redelivery for the rest of the TTL:
- The hand-off task releases the claim on cancellation too (stop() drains it
while it resolves the connection identity), via await_drained so the
cancellation still propagates.
- ChannelManager._discard_pending_inbound releases the key of a claimed
message that shutdown drops from the queue, after the grace period and on
a cancelled shutdown.
- The adapter tracks claims until they are committed. A release refused
because stop() already closed cross-loop intake, or a hand-off cancelled
before it ran, leaves the claim tracked, and stop() releases it last.
* fix(gateway): keep IM run idempotency stable when /new rebinds the conversation
Review round 2 on #7104. The Gateway scoped every Idempotency-Key by owner
and thread. /new rebinds an IM conversation to a fresh thread, so a late
redelivery of a message admitted on the old thread was requested on the new
one, got a different scoped key and started a second run.
- Keys from trusted internal callers (the IM channel manager) are scoped by
owner only (`internal-run:` prefix); browser and API keys keep the
owner-and-thread scope (`http-run:`).
- A key that resolves to a run of another thread or user raises
RunIdempotencyKeyMismatch, a RuntimeError subclass so in-process launchers
see the same type, and start_run answers it with 409 "Idempotency-Key
already used with a different request" instead of a 500. The channel
manager already treats that 409 as the message's admitted run.
Docs: API.md, IM_CHANNEL_CONNECTIONS.md, gateway AGENTS.md (reworded to stay
under its budget), CHANGELOG (en/zh).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(multi-instance): publish skill_manage cache resets and tolerate a concurrent namespace create
Two leftovers from the multi-instance work (#7097).
The agent's skill_manage tool refreshed only its own process's skills
prompt cache after a create, edit, patch, delete, write_file or
remove_file, so a Gateway replica sharing the home volume kept rendering
that user's previous skill list. Every mutation now runs one drained tail,
as the skill routes do: the write and its history record, the local
refresh, then publish_skills_cache_reset(user_id=...) off the event loop.
A run cancelled while the write is in flight still refreshes and
publishes before the cancellation propagates. A marker that cannot be
written is logged inside the tail and raised as SkillCacheResetPublishError
with a message saying the change was applied, so the model does not retry
a write that succeeded.
Two provisioner replicas starting together against a missing
K8S_NAMESPACE both read 404 and both create; the loser's 409 escaped the
lifespan and crashed it. A 409 on create now counts as the namespace
existing, as the Pod and Service creates already do.
Also drops a stale duplicate of the multi-instance "declaration also
drives" note in CONFIGURATION.md.
* docs(changelog): reference #7099 for the multi-instance leftovers
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): share encrypted runtime channel config across Gateway instances
IM bot credentials entered in the web UI lived in each host's plaintext
channels/runtime-config.json, and the router cached the merged config on
app.state for the life of the process. With several Gateway instances a
save or disconnect served by an instance that does not hold the
provider's leader lease never reached the transport holder: it kept the
old credentials until it next started the channel, and a disconnect
could leave the old bot connected.
- New shared table channel_runtime_configs (migration 0044): one row per
provider with the config block encrypted under DEER_FLOW_CREDENTIALS_KEY,
a plaintext disconnect tombstone, and a version bumped by every save.
- SqlChannelRuntimeConfigStore replaces the JSON file for sqlite/postgres
(memory keeps the file); it is synchronous on app_sync_sqlalchemy_url
because its callers already run in worker threads.
- The router re-reads the store on every request instead of caching the
merged config, and refuses (503) to save credentials without a key.
- The leader-lease supervisor compares the versions on each renewal tick
and restarts or stops the holder's transport when a provider's config
changed; a takeover reads a disconnect instead of reviving cached
credentials.
- The first start imports an existing runtime-config.json (rows already
in the table win) and deletes the plaintext file.
* docs(changelog): reference #6988 for the shared runtime channel config
* ci: rerun checks after #6999 fixed the frontend guidance budget on main
* fix(channels): start a held transport that is down when its runtime config changes
When the lease holder's transport was down (a start that failed on the
old credentials), applying a peer-saved config only updated the cached
config, and the transport started on the next lease round: two renewal
ticks instead of the one the docs promise. The apply step now starts it
on the held lease right away, with the lease round's backoff bookkeeping.
* feat(channels): generalize IM group targets and add task/goal run notifications
- Generalize proactive group notification capability (proactive_group_notifications), binding validation, and scheduled task group selection across Feishu, QQ, and WeCom while keeping WeChat personal-only.
- Add QQ group active notification delivery via POST /v2/groups/{group_openid}/messages (msg_type=2) and WeCom group active notification delivery via WS Client.sendMessage(chatid, body) with bot_id:group:chatid workspace isolation.
- Generalize notification outbox sources via migration 0043_runtime_im_notices (source_kind, source_key, run_id, nullable task_id/task_run_id) with backward-compatible SQLite/PostgreSQL upgrades.
- Add opt-in ordinary task & goal run IM notifications (run_completed, run_failed, human_input_requested) on the shared outbox, delivery worker, renderer, and settings UI.
* fix(notifications): address review feedback and group/runtime notification edge cases
* test(migrations): update CURRENT_HEAD to 0043_runtime_im_notices across migration chain tests
* fix(channels): preserve group binding names and legacy authorization
Carry owner-chosen group names through existing connect state, keep private bindings out of legacy group fallback, and restore completion metadata retries. Cover real binding and worker finalization paths and synchronize notification docs.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep IM runs alive when the creating request disconnects
The channel manager holds a `runs.wait` / `runs.stream` request open for
the whole run, and that request dies with the Gateway instance that
received the IM message: a leader hand-off, a rolling restart, a crash.
The Gateway cancels a run whose creating request disconnects unless the
request asked for `on_disconnect: continue`, so every IM run in flight
was cancelled on failover.
Channel runs now ask for `continue`, like scheduled-task and MCP-task
runs already do. The run finishes on the Gateway executing it and its
thread state is persisted, so the answer stays readable in the web UI
even though the IM reply is lost with the old transport. The key is set
after the fire-and-forget branch because the SDK's `runs.create` has no
`on_disconnect` parameter.
Docs: CONFIGURATION.md and the Helm/deployment guides no longer list
"failover cancels IM runs" as a known limit; IM_CHANNEL_CONNECTIONS.md
and the channels AGENTS.md describe the policy.
* docs(changelog): link #6876
* fix(channels): keep the inbound dedupe key once a run may be executing
With `on_disconnect: continue` a run outlives the manager worker that
requested it, but the worker loop released the inbound dedupe key on
cancellation, so a provider redelivery on the replacement instance
admitted the same event again and executed the turn twice (review on
#6876).
Each inbound message now carries an admission record: `requested`
flips before the run request leaves the manager, `confirmed` once the
Gateway proved it admitted the run (a run id, a stream frame, a
completed response). The generic release paths (worker cancellation,
worker/handler exceptions, a broken stream) keep the key when the run
may continue: after confirmation, on cancellation after the request
was sent, and on post-request transport failures (read timeout, read
error, protocol error). Pre-request failures, Gateway rejections
(busy thread, stale pin, status errors, connection failures) and the
stream's terminal `error` frame still release it so the redelivery
can retry.
* feat(channels): arbitrate IM transports with a per-provider leader lease
Every Gateway instance connected every enabled IM platform with the same
bot credentials, so channels could not run on more than one instance
(Telegram 409 storms, Discord answering twice, scattered per-process
state). channels.leader_lease was reserved by #6746; this implements it.
- channel_leases table (migration 0041): one row per provider. Acquire is
one dialect-native upsert decided against the database clock and bumps
a fencing epoch; renew extends only a live lease at the holder's epoch;
release sets the expiry to 0 so a peer takes over on its next tick.
- app/channels/leadership.py: the holder's local monotonic deadline
(send time of the last confirmed renewal + lease_seconds); holds() turns
false one renewal interval early, so the transport stops at least
grace_seconds before a peer may connect. DB calls are bounded.
- ChannelService: start fence in _start_channel (no DB call) behind every
start path; fixed-rate supervisor renews/acquires and runs transport
starts/stops as per-provider tasks; takeover reloads the provider's
saved config; failed starts back off; shutdown releases only leases
whose transport stopped. get_status reports leader/leader_here and the
holder's transport state on every instance.
- Notification worker leaves rows of providers another instance holds.
- Gate: the enabled=false refusal now recommends auto/true; an explicit
true with database.backend memory is refused when a channel is enabled.
- Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated.
* docs(changelog): link the channel leader lease entry to #6843
* fix(channels): see peer-held leases for providers absent from local config
Review on #6843: a provider connected from the web UI on a peer never
enters this instance's channel config, so the notification worker here
still claimed its rows (and parked them for 900s) and GET /api/channels/
reported no leader for it. Every supervision round, and startup, now also
reads the whole lease table once (bounded), and notification exclusions
and status use those observations: a provider whose live lease a peer
holds is left to it even without local config.
Running that read concurrently with renewals exposed that, through
SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on
another connection can hand back one of its own bind values instead of
the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do
not show it). A lease must never take such a row for a confirmed
renewal, so acquire/renew/release now decide by affected-row count and
acquire reads its epoch back in the same transaction.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): add channels.leader_lease and gate it under multi-process signals
IM channels other than GitHub connect outbound to their platform. With more
than one Gateway instance each one opens that connection with the same bot
credentials: Telegram rejects the second poller with 409, Discord answers every
message twice and per-process ChannelManager state diverges. The multi-process
startup gate said nothing about channels; the only guard was a comment in the
Helm values.
Add the `channels.leader_lease` section (`enabled: auto|true|false`,
`lease_seconds`, `grace_seconds`; startup-only, registered in the reload
boundary) that the per-provider leader lease will read, validated by a Pydantic
model from the non-schema `channels` mapping. The gate now refuses
`enabled: false` under a multi-process signal while a non-GitHub provider is
enabled under `channels.*` or connectable through `channel_connections`, naming
the providers and the rollback; a malformed block is refused on every topology.
`auto` follows the multi-process signal, so a single instance is unchanged.
ChannelService strips the block before its per-provider loops and exposes it
as `leader_lease`; `ensure_ready_channels` only visits registered providers.
The two-Gateway harness overlay leaves the block alone. config_version 60 in
the example, the Helm chart and its README; docs replace the comment-only
"keep 1 replica" guidance with the gate.
* docs(changelog): link #6746
* fix(channels): stop recommending the unimplemented leader lease as a remedy
Review on #6746: the refusal told operators to switch channels.leader_lease
from false to auto/true "so one instance owns each provider's connection", but
the transport arbitration ships separately, so that change only bypassed the
clause and every worker still started the same bot.
The refusal now offers only the remedies that work today (a single instance, or
disabling the channels) and states that auto/true are reserved for the lease
that ships separately. A regression helper in the gate tests asserts the
remedies start with the rollback and that auto/true are never offered. The
config example, Helm values/README, CONFIGURATION.md, README, the deployment
guide (en/zh), the model docstrings and both changelogs describe auto/true as
reserved settings rather than an ownership guarantee.
* fix(channels): warn on an enabled unknown channel key; reword the changelog entry
Review on #6746: the registry guard in ensure_ready_channels skipped a
misspelled provider key (channels.telegam) silently, where it previously
reached _start_channel and logged 'Unknown channel type'. An enabled dict
block that is neither a registered provider nor leader_lease now logs a
warning that lists the registry.
The changelog entry claimed the chart and docs drop the "keep 1 replica"
guidance; they keep it (auto/true do not make several replicas safe yet) and
only stop relying on the comment alone.
* fix(dev): disable UI channel providers in multi-instance overlay
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add You.com web search provider
Adds You.com as an opt-in community `web_search` engine, mirroring the
existing community engines (tavily, brave, exa, firecrawl, groundroute).
You.com runs its own web index and returns extracted page snippets per
hit rather than a single meta description, so a research loop gets usable
context without a follow-up fetch.
The provider picks its endpoint by whether a key is configured, so it
works out of the box and scales up when the operator adds one:
- no key -> /v1/agents/search, keyless free tier
- YDC_API_KEY -> /v1/search with X-API-Key, full Search API
Endpoint and headers are always chosen together; the keyless endpoint
rejects an auth header, so a stale key can never be sent there. Both
return the same body shape, so one normalizer covers both.
httpx only, no new dependency and no pyproject extra, so uv.lock is
untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint
* fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY
The doctor detail returned before any key check, so operators who had
set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'.
Reworded to match the sibling free-provider entries and stay accurate in
both states.
* fix(search): sweep youcom max_results onto the shared owner
The single-owner gate that landed with #6563 flags this branch's private
_coerce_max_results as a newly hand-rolled copy, so backend-unit-tests
now fails after merging main. Follow the coordination note from review:
delegate to deerflow.community.search_max_results.coerce_max_results,
which owns the rejection profile (booleans, non-integral floats,
OverflowError), and keep only the 1-100 mirror of You.com's server-side
count clamp on top — clamping policy stays per-provider.
The .inf regression test now goes through the public tool path and also
pins the bool and non-integral-float shapes of the shared bar.
* docs(tools): use the bare you.com platform link in config and docs
Match every other provider entry: no other provider in config.example.yaml
or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params
from the three you.com/platform links. Integration attribution stays in
the identifying User-Agent sent to You.com hosts.
* test(youcom): cover shared max_results ownership and bounds
Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters.
---------
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(dev): add a two-Gateway multi-instance harness script
There was no way to run the deployment.multi_instance topology locally:
scripts/serve.sh pins one Gateway to port 8001, the local nginx config has a
single upstream, and getting two processes past the multi-instance startup gate
needs Postgres, Redis, a shared DEER_FLOW_HOME, shared auth secrets and a
per-process DeerMem retrieval index by hand.
scripts/dev_multi_instance.sh (make dev-multi / dev-multi-check /
dev-multi-down) starts throwaway Postgres and Redis containers on loopback,
renders a config from the developer's config.yaml (or config.example.yaml),
and runs Gateway A on 8001 and Gateway B on 8011 with one shared home, one
copy of extensions_config.json and generated AUTH_JWT_SECRET,
DEER_FLOW_INTERNAL_AUTH_TOKEN and DEER_FLOW_CREDENTIALS_KEY values, plus an
optional round-robin nginx on 2027. B starts after A answers /health: Alembic
migrations are advisory-locked, but the LangGraph checkpointer/store setup()
that follows is not.
scripts/dev_multi_instance.py holds the testable parts: the config overlay
(neutralizing and reporting settings the gate refuses), the secrets file, the
nginx config, and `check`, which automates the scriptable cross-instance
acceptance checks: readiness with stream_bridge ok, A-issued sessions and the
internal token accepted by B, thread/upload/artifact visibility, a skill
toggle seen by B, nginx balancing, and SSE join on B with Last-Event-ID resume
on A. Tests pin that the rendered config passes the real
_enforce_postgres_for_multi_worker gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link the two-Gateway harness entries to #6613
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(dev): keep the multi-instance harness inside its own state
Address review findings on the two-Gateway harness (#6613):
- Containers carry a deerflow.harness.state-dir label; up/down remove only
containers of their own state dir and refuse those of another checkout or
DEERFLOW_MI_STATE_DIR instead of force-removing them.
- Containers are removed with `docker rm -f -v`, so the anonymous volumes
the postgres and redis images declare no longer leak on every up/down.
- `up` installs an EXIT trap once it starts containers, so any later failure
(including B's port being taken after preflight) tears down A and both
containers; start_gateway returns failures instead of exiting the script,
and the Gateway is disowned so stopping it prints no job notice.
- The overlay removes any checkpointer section (a postgres one would write
checkpoints and the Store into the developer's database), points an
explicit sandbox.ownership and a Redis checkpoint_cache at the harness
Redis (keeping ownership timing), drops DeerMem storage_path (both
spellings) and the blob_storage root so data lands in the harness home,
gives a local AIO sandbox a state-dir-specific container_prefix, and
reports a non-DeerMem memory backend it keeps sharing.
- nginx paths are rendered as quoted string literals ('$' and control
characters are refused), so a state dir with spaces works.
- `check` skips the run-based SSE check when model discovery did not
succeed, unless --with-llm, and enforces the SSE time budget per raw line
so heartbeat-only traffic cannot hold it open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(scripts): keep the harness guidance under the AGENTS.md soft limit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Fail closed on delegated fetch backends to close SSRF gap
* fix(community): resolve delegated-fetch credentials from one config snapshot
Address review feedback on the fail-closed SSRF guard:
- firecrawl/fastcrw client construction no longer re-reads get_app_config(),
so the API key and endpoint come from the same revision the endpoint screen
validated (prevents cross-revision credential misrouting).
- fix the browserless test double to accept the new two-argument
_get_browserless_client signature.
- document the check-only (DNS rebinding) caveat on
validate_delegated_backend_url.
- add a CHANGELOG upgrade note for the default self-hosted backend break.
* fix(doctor): consolidate delegated fetch backend checks and deployment guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): share chat-to-thread bindings through the database
ChannelManager kept its IM chat-to-thread bindings
(channel_name:chat_id[:topic_id] -> thread_id) in a per-process JSON file,
channels/store.json, loaded once and rewritten whole on every change. With
several Gateway processes a binding created on one replica was invisible to
the others (the next message for the same chat opened a second thread) and
concurrent writers clobbered each other's file.
- New channel_thread_bindings table (migration 0036_channel_thread_bindings)
and harness package deerflow.persistence.channel_thread_bindings: key PK
(the legacy composite string), channel_name, chat_id, nullable topic_id,
thread_id, user_id, epoch created_at/updated_at; index (channel_name,
chat_id). Dialect upsert keeps created_at; prefix remove and batched
ON CONFLICT DO NOTHING import.
- app.channels.store.ChannelStore is an async protocol: SqlChannelStore for
sqlite/postgres, JsonChannelStore (file I/O via asyncio.to_thread) for
database.backend: memory. resolve_channel_store picks at ChannelService
construction; ChannelService.start() imports an existing store.json once
into an empty table and renames it store.json.migrated (a concurrently
importing peer neither duplicates nor clobbers; a populated table leaves
the file alone).
- Callers updated: manager, connection_identity.lookup_thread_id, Feishu
(send() awaits the mapping writer; the synchronous lark callback bridges
its two store calls to the Gateway loop with a bounded wait), Discord
(channels_dir).
- Head pins in the 0027_0028/0029/0032/0034/0035 migration tests point at
the new head; docs (channels and migrations AGENTS.md, CONFIGURATION.md,
README) and CHANGELOG updated.
* docs(changelog): link the channel thread bindings entry to #6558
* fix(channels): reject only the Gateway loop in the Feishu store bridge
lark-oapi 1.5.5 invokes the message callback synchronously from its async
frame handler, on the private event loop that FeishuChannel._run_ws starts
for the SDK thread. The bridge's guard treated any running loop as unsafe,
so every real inbound message skipped the stored-topic lookup and the
pending-mapping restore: a reply whose parent card was mapped but whose
root was not fell back to the unknown root, and could consume an
unrelated pending clarification.
Only a wait issued from the Gateway loop itself (the loop that executes
the submitted coroutine) is refused now; the SDK's own loop may block for
the bounded wait, as that thread blocked on the JSON rewrite before.
Tests drive _on_message from a worker thread running its own loop, the
production shape, for both the lookup and the write path, and pin the
Gateway-loop refusal.
* fix(channels): defer unresolved Feishu lookups and harden the store.json import
Review follow-ups on the chat-to-thread binding store (#6558):
- Import: a legacy entry whose key, channel_name, chat_id, topic_id,
thread_id or user_id exceeds its column width is skipped and counted
with the malformed ones instead of failing the whole batch on
PostgreSQL (SQLite silently accepted the same row, so the backends
diverged on one file). The corrupt-file warning now says what each
caller does: the JSON store starts fresh, the import is skipped and
the file kept.
- Feishu bridge: store calls run on the Gateway loop under a 3s
asyncio.wait_for (the SDK loop also services ws keepalives; the old
10s bound could stall it per message), and the lark-thread wait is
that budget plus 1s. A deadline or store error is StoreLookupUnresolved,
never a miss: _on_message no longer routes such a reply onto its
unknown root (which created a second thread and later overwrote the
parent alias). The message joins a per-chat FIFO drained on the
Gateway loop with backoff for up to 30s, then is dropped with a
warning. _on_message is split into parse / resolve / dispatch so the
deferred path dispatches the same way; later messages for the chat
queue behind the deferred one.
* fix(tests): expect follow_redirects in the Jina client kwargs after #5808#5808 made JinaClient pass follow_redirects=True to httpx.AsyncClient and
updated test_jina_client.py, but test_jina_response_limit.py and
test_jina_retry_after.py still pin the exact client kwargs to proxy and
trust_env, so every backend-unit-tests shard that lands on them fails on
main (the backend unit-test workflow only runs on pull requests, which is
why the push to main did not surface it). Update the two expectations to
include the new keyword.
(cherry picked from commit d5aa23156ce19ce36d95df03953442f83ae6d050)
* Revert "fix(tests): expect follow_redirects in the Jina client kwargs after #5808"
This reverts commit 5bee00bfb09c65a30f77fb4df47a1f82c47ecef7.
* fix(channels): bound deferred Feishu routing by the inbound queue capacity
Review follow-up on #6558: a store outage let every subsequent message for
a chat join the deferred queue with no capacity check (the bus reservation
was only taken after routing succeeded), and each queued message started a
fresh 30s retry budget when it reached the head, so a burst could be
retained for N x 30s and grow without bound.
- Intake is reserved BEFORE routing for every store-backed message
(bus.reserve_inbound is thread-safe and already used from SDK threads),
so deferred and live messages share the one configured bound,
channels.inbound_queue_maxsize; a message that cannot be admitted is
dropped at once with a warning, never queued or routed as a miss.
- The reservation travels with the message into dispatch (committed by
_prepare_inbound), is released on the batch path (the flush reserves at
publish time), on drop after the budget, on a failed deferral, on
dispatch failure and for every leftover when the drain task is cancelled
by stop().
- The retry budget runs from the message's arrival: a queued message whose
budget is spent when it reaches the head is dropped without a lookup.
- The bounded resolver returns the unresolved outcome instead of raising
it through the tracked task, which logged an ERROR for an expected,
handled store timeout.
- Upstream's new blocking-io test constructs the store by its old class
name; it now uses JsonChannelStore.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(auth): count login lockouts in the database across Gateway replicas
POST /api/v1/auth/login/local throttled failed logins per client IP in a
module-level dict, so with N Gateway replicas behind one load balancer an
attacker got N x auth.local.max_login_attempts guesses and a lockout on
one replica was invisible to the others.
Move the counter behind a LoginThrottleStore (deerflow.persistence.
login_throttle): MemoryLoginThrottleStore keeps the historical per-process
semantics and 10k tracked-IP bound; SqlLoginThrottleStore shares one
login_throttle row per IP through the application database, incrementing
and starting the lock in a single dialect-native upsert, applying check
decisions with compare-and-set predicates in a separate write
transaction, and sweeping served locks plus 24h-idle counters in bounded
batches. Both stores pin the existing contract: the lock starts when the
count reaches max_login_attempts, the committed duration is honored when
the policy shrinks mid-lock, a served sentence is never resurrected,
tightening keeps the count, success resets the IP, and the 429 status and
message are unchanged.
New startup-only selector auth.local.throttle_storage (auto | memory | db,
default auto = the database whenever database.backend is sqlite or
postgres). The Gateway installs the store once at startup after the
persistence engine; deps._validate_login_throttle_storage warns when a
multi-process deployment keeps memory. Migration 0034_login_throttle
creates the table; config_version 55 -> 56 in config.example.yaml and the
Helm chart. Docs: README, CONFIGURATION.md, AUTH_DESIGN.md, config and
migrations AGENTS.md, CHANGELOG.
* docs(changelog): link the login throttle entry to #6501
* fix(auth): re-check expiry on the row the login-throttle sweep deletes
The bounded cleanup in SqlLoginThrottleStore.record_failure deleted
`WHERE ip IN (SELECT ip ... WHERE <served or stale> LIMIT 200)`. On
PostgreSQL READ COMMITTED the expiry predicate applied only inside the
snapshot-bound subquery: when the DELETE had to wait on a row lock held
by a concurrent check() that was extending the same lock under a raised
lockout_seconds, PostgreSQL re-evaluated only the outer WHERE on the
updated tuple, IP membership still held for the old candidate, and the
now-active lock was deleted; the next attempt from that IP got a clean
record and reached password verification (review of #6501, P2).
Repeat the served/stale predicate on the DELETE target as well
(SqlLoginThrottleStore.sweep_statement), keeping the bounded candidate
subquery, so the re-evaluation sees the updated row's expiry. The other
writes already carry their full compare-and-set predicate on the target
row: _discard and the check() UPDATE both use _matches (ip, fail_count,
locked_at, lock_duration_seconds), so a tuple changed while they waited
no longer matches and the stale decision is skipped. _discard keeps the
equality compare-and-set rather than adding an expiry condition: its
early-release branch deletes a lock whose committed sentence has not
elapsed under the lowered live policy, which an expiry predicate would
wrongly block, and the equality predicate already pins every input of
the decision. reset stays unconditional by design.
Tests: a dialect-compile test asserts the DELETE's own WHERE carries the
expiry predicate for postgresql and sqlite (red on the previous HEAD), and
a PostgreSQL-only test (skipped without TEST_POSTGRES_URI) reproduces the
update-versus-sweep interleaving with two engines, waiting on
pg_stat_activity for the sweep to block before the extender commits.
Docs: sql.py concurrency contract and the migrations guide's 0034 bullet.
* fix(auth): keep an active login lock's clock and fail closed on explicit db
Address the "changes requested" review on #6501.
Lock clock (confirmed defect): record_failure re-stamped locked_at = now
for every failure on an already-locked row, so a sustained attacker's
sentence slid forward with each attempt. Both stores now apply one rule
in one atomic decision: new_count = fail_count + 1; below max_attempts the
IP counts with no lock (a failure after the operator raised the threshold
clears an existing lock); at or over it, a failure during an *active* lock
keeps locked_at and the committed lock_duration unchanged, while a
never-locked or served row starts anew with now / lockout_seconds (so the
failure that reaches a raised threshold starts a fresh lock rather than
keeping a stale locked_at). The SQL upsert expresses it as nested case()
on the row's own values; the memory store mirrors it. The sweeps in both
stores now leave the row being recorded to that decision instead of
deleting a served lock or stale counter first, which split the contract
(SQL would have produced a fresh (1, NULL, NULL) row). base.py states the
rule.
Explicit db downgrade: resolve_login_throttle_store raises SystemExit —
the failure type of the other startup validations in app.gateway.deps —
when throttle_storage='db' resolves to the shared table but no engine is
initialised, so langgraph_runtime refuses to start instead of silently
running every replica on per-process counters; the lazy bare-app path
applies the same rule and installs nothing. 'auto' keeps warning and
falling back. 'db' on a memory database still degrades with a warning
(there is no shared table to insist on).
Non-blocking items: _key() logs a warning when a client key is truncated
to 255 characters (never the key itself); the sql.py docstring covers
wall-clock locked_at across replicas (run NTP; skew shifts evaluation by
that much) and why STALE_COUNTER_SECONDS=24h / SWEEP_BATCH_SIZE=200 were
chosen; a store-level blocking-I/O anchor covers the SQL reset path; the
_validate_login_throttle_storage warning was verified to render
'throttle_storage=memory' for the StrEnum (the gate test now passes the
enum member and asserts that text). reset stays unconditional: a peer
failure that commits after a reset leaves fail_count=1, one genuine
failure after the cleared history, which a new test documents.
Docs: CHANGELOG entry, config.example.yaml, CONFIGURATION.md,
AUTH_DESIGN.md and the throttle_storage field description.
* fix(auth): re-evaluate a login-throttle check whose compare-and-set missed
SqlLoginThrottleStore.check applied its decision (commit the live
duration, or clear a served / released lock) with a compare-and-set
predicate on the snapshot it had read, but ignored the affected-row
count. When a peer changed the row between the read and the write — an
admitted wrong password incrementing fail_count on another replica, or a
successful login clearing the row — the UPDATE or DELETE matched zero
rows, yet check reported the decision as applied: it could return an
extended 60s sentence while the row kept its 1s duration, so the next
check cleared the lock and let password verification through; a live
duration decrease could be dropped the same way; and a row a peer had
just cleared was still reported as locked (#6501 review, P2).
Every compare-and-set write in check now checks result.rowcount. On a
miss the row is re-read and re-evaluated from the top, up to
CHECK_CAS_ATTEMPTS (3) times, applying a fresh compare-and-set decision
on the new snapshot and never clobbering a reset or replacement: a row
that is gone is reported as allowed, a fresh peer lock is reported as
locked, and an incremented count gets the extension or decrease
committed on it. Exhausting the bound fails closed: nothing is written
and the answer is the last snapshot's lock under the live policy, logged
at debug. The read stays in its own short transaction and each write in
a separate one.
The evaluation is now one shared pure function, evaluate_check in
base.py (CheckDecision), used by both stores so the contract cannot
diverge. The memory store keeps a single pass: its compare-and-set
cannot miss because there is no await between the read and the write on
the loop thread, which its docstring now states.
Tests (red first on df26c2acf) use a session proxy that runs the peer
mutation on a second engine right after the snapshot read ends its
transaction: the reviewer's raised-duration interleaving (59.5s reported
and committed, the increment survives, a later check still locked), the
mirrored decrease, early release on the fresh row, a reset landing
between read and write (0.0, no row recreated), and the bound-exhausted
fail-closed case.
* docs(changelog): resolve merge markers left in the Unreleased section
* test(migrations): pin 0035_login_throttle as the head in the 0034 watermark test
Upstream #6477 added test_migration_0034_run_event_seq_watermark.py asserting
0034 is the single head; this branch chains 0035_login_throttle after it, so
the test now pins the current head and walks the chain like its siblings.
* perf(auth): one session per login-throttle check, one transaction per failure
Two non-blocking performance suggestions from the #6501 review.
Single session per check. The router probed the row with store.get()
and then called store.check(), which re-read it: a recorded IP cost two
connection-pool checkouts and two SELECTs per request on the
unauthenticated login path, letting a lockout flood compete with
legitimate traffic for Postgres connections. LoginThrottleStore.check
now takes the policy as an async callable (LoginThrottlePolicy) and
resolves it lazily inside the store: the probe, the policy resolution
and the compare-and-set loop share one session, a clean IP returns 0.0
without ever calling the policy (the config.yaml read is still skipped
for the common case), and a recorded IP resolves it exactly once, also
across compare-and-set retries. The decision is made on a snapshot read
after the resolution: the policy read is a yield point, and a decision
that needs no write has no compare-and-set to catch a row a peer changed
meanwhile — so one session, but still a read after the yield. The
memory store takes the same signature and re-reads its record after the
await for the same reason. The router passes its asyncio.to_thread
policy read as the callable and drops the separate probe; get stays on
the protocol for tests and anchors. No short-TTL verdict cache: it would
mask a successful login's reset for the TTL window, and the single
session already halves the hot path.
Sweep in the upsert's transaction, plus an index. record_failure ran the
housekeeping DELETE in its own transaction and the upsert in a second
one, two commits (two WAL flushes on PostgreSQL) per failed login; the
sweep now rides in the upsert's transaction with one commit, the keep
exclusion already protecting the row being recorded. The new
ix_login_throttle_updated_at (ORM __table_args__ and the unreleased
0035 migration, created after the table and dropped before it, each
guarded separately so a table left by an interrupted attempt still gets
its index) serves the sweep's stale-counter predicate; the served-lock
predicate scans a table the sweep itself keeps small.
Tests (red first on a137fbc47): a clean IP never resolves the policy and
a recorded IP resolves it exactly once (both stores), the SQL store opens
exactly one session per check and commits once per record_failure
(counting wrappers), the migration test asserts the index after upgrade,
after completing a partial upgrade that lacked it, and gone after
downgrade while compare_metadata stays empty; the router-level
"mutated during policy resolution" tests now inject their mutation
through the policy callable.
* docs(changelog): note config_version 57 in the login throttle entry
* fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers
DeerMem's derived SQLite FTS5 index for every user was one WAL database pinned
to {storage_path}/.retrieval. With several Gateway Pods sharing one home volume
each Pod opened that file over a network filesystem (unsupported by SQLite),
every Pod start emptied and refilled it under its peers, one Pod's corruption
recovery deleted files the others held open, and a Pod never learned that a
peer had written a user's facts, so memory_search served stale results.
- New backend-private key memory.backend_config.retrieval_index_path: empty
keeps {storage_path}/.retrieval, a relative path resolves against
storage_path, an absolute path is used as-is. The full rebuild and the
corruption recovery act only on that configured index.
- rebuild_index records each agent scope's manifest signature
(mtime_ns, size, revision) before reading its facts; search_facts rebuilds a
scope whose live signature differs because another process wrote the user's
memory. An own commit advances the recorded signatures of that user's scopes
only when they were in sync at the previous revision, so own writes never
rebuild while an interleaved peer write still does.
- Gateway startup warns, in the style of _validate_agent_storage, when a
declared multi-instance deployment keeps the DeerMem index inside
storage_path.
- config.example.yaml documents the key; config_version 55 -> 56, mirrored in
the Helm chart and its README.
- Helm mounts an emptyDir at /var/lib/deerflow/memory-index, points the key at
it, drops the legacy memory.storage_path: memory.json line, and documents the
memory layout next to the other multi-replica prerequisites.
* docs(changelog): link the memory retrieval index entry to #6494
* fix(memory): never advance the index signature on a no-op commit and resync whole scopes
Two review findings on the cross-replica retrieval re-sync:
- save() and apply_changes() inferred the replaced manifest revision as
"returned revision - 1". The commit helper returns the existing manifest
unchanged for a no-op (unchanged-value or empty change set), so after a peer
bumped the manifest an own no-op write advanced this process's stale index
signature to the peer's revision without carrying the peer's delta: a fact the
peer deleted stayed searchable until the manifest changed again. Both paths
now read the real pre-commit revision under the user lock and pass
committed=(pre_revision, signature) only when the commit produced a new
revision; a no-op passes None so the next search still rebuilds the scope.
- The scoped rebuild (signature mismatch and the pre-existing dirty-scope path)
listed facts through list_facts() with its default limit of 100 and then
replaced every row of the scope, so a scope with more than 100 facts lost its
tail on the first re-sync. It now reads the scope's complete fact list.
* fix(memory): fence index signature promotion against a concurrent scope rebuild
Review finding on the cross-replica re-sync: a successful adapter notification
does not prove its delta is still in the index when the signature is promoted.
The SQLite lock covers single adapter calls only, and neither the scope lock
nor DeerMem's warm-up lock spans "notify, then promote", so rebuild_index can
replace a scope's rows in between. After a peer summary-only commit (r+1), A's
search starts a scoped refresh that captures the r+1 signature and reads f1/f2;
A's own deletion of f2 commits r+2 and removes the row; the refresh then
installs its stale f1/f2 and marks the scope synced at r+1; the deletion
resumes and promoted that signature to r+2, so later searches saw the live
signature as equal and kept returning the resurrected f2 until another manifest
change.
Mechanism: a generation fence rather than a lock. rebuild_index bumps a
per-scope generation (scoped rebuild) or a storage-wide generation (full
rebuild) under _cache_lock whenever it publishes or forgets a scope's rows, in
both mark_synced and forget_synced. _dispatch_retrieval_notifications snapshots
the generations under _cache_lock before its first adapter call and, at
promotion time, skips every scope whose generation moved (and everything when
the storage-wide generation moved), leaving the rebuild's own captured
signature in place; that signature predates the commit, so the next search's
comparison rebuilds the scope. A lock would have had to serialize notification
plus promotion with bulk install plus publication while search_facts ->
rebuild_index -> load() already takes the scope, file and cache locks; the
fence adds no lock-order obligations and only costs a redundant rebuild in the
rare race. Every interleaving is covered: a rebuild that completes before the
snapshot is reflected in the captured generation and promotion proceeds on an
index that carries both the rebuild's rows and this write's delta; one that
publishes between snapshot and promotion is skipped; one that publishes after
the promotion records its stale captured signature, which the next search
catches.
Tests reproduce both orders deterministically with threading.Event ordering
(adapter wrapper that holds the bulk install and the remove notification):
test_promotion_is_fenced_against_a_refresh_that_installs_in_between (red before
the fix) and
test_stale_refresh_installing_after_a_promotion_is_caught_by_the_next_search
(regression guard for the mirrored order).
* fix(memory): publish a rebuild's signature under the lock that replaced its rows
Review finding on the cross-replica re-sync: two peer-triggered refreshes of
one scope could replace rows in one order and publish their captured
signatures in the other. On a warmed Pod A, refresh R1 captures r+1 with f1/f2
and waits before installing; a peer deletes f2 at r+2; refresh R2 captures
r+2, installs f1 only and pauses after bulk_rebuild returns, before
mark_synced; R1 installs its stale f1/f2 and publishes r+1; R2 then publishes
r+2 over R1's rows. Later searches saw the live signature as equal and kept
returning the deleted fact. The generation fence only guards own-write
promotion, and the adapter's SQLite lock ends when the install returns.
Mechanism: a per-storage _retrieval_publish_lock held across the row
replacement and the signature publication, for the bulk path (bulk_rebuild +
mark_synced / forget_synced) and the clear+upsert fallback, scoped and full
alike. Overlapping refreshes of a scope therefore serialize their
install+publish pairs: whichever installs last also publishes last, and a stale
captured signature published last differs from the live manifest, so the next
search's compare rebuilds the scope. Fact reading (load() / the glob) stays
outside the lock so a slow read never blocks refreshes of other scopes. The
startup full rebuild holds the lock for its single bulk install; scoped
refreshes wait for it, which is acceptable because before warm-up completes a
search would be rebuilding its scope anyway (DeerMem.warm_retrieval takes its
own _retrieval_lock only after rebuild() returns, so the two never nest).
Lock order: the publish lock is acquired with no storage lock held --
search_facts releases _cache_lock before calling rebuild_index, reload() leaves
its scope/file-lock block before rebuilding, and DeerMem._ensure_retrieval_scopes
holds only its outer RLock, which nothing inside rebuild_index acquires. Inside
the lock only the adapter's own lock and _cache_lock are taken, and both are
leaf locks that never call back into storage, so no cycle is possible.
Test: test_rebuild_publication_stays_ordered_with_row_replacement drives the
reviewer's interleaving with threading.Event ordering (red before the fix). The
pause harness now holds a refresh after its fact read, outside the publish
lock, instead of inside the adapter's rebuild, so the earlier fence tests keep
their semantics without holding the new lock while paused.
* fix(memory): read each manifest once per full rebuild and serve the local index on a compare read error
Two suggestion-level review items on the cross-replica retrieval re-sync.
- The full rebuild cached manifest signatures keyed by (manifest path, agent
name), but _scope_signature reads only the manifest file -- every agent
bucket of a user shares one memory.json -- so the startup scan stat+read+
parsed the same file once per bucket that has facts. The cache is now keyed
by the manifest path alone while synced signatures are still recorded per
(user, agent) scope: users manifest reads instead of users x agents.
- The per-search freshness compare put a shared-volume manifest read on every
search_facts call, and _load_memory_file raises MemoryStorageCorruption on
OSError and malformed JSON, so a transient network-filesystem read error or a
corrupt memory.json failed a search the Pod-local index could still answer.
The compare now catches OSError, ValueError and MemoryStorageCorruption: it
logs a warning naming the scope (traceback at debug), serves the current
local index for that scope and leaves the recorded synced signature
untouched so the next search compares again. It deliberately does not mark
the scope dirty: a scoped rebuild would hit the same failing read, forget the
scope's signature and fail the search path too.
* docs(changelog): resolve merge markers left in the Unreleased section
* fix(memory): forget a scope's index signature when an own delta cannot be proved compatible
Review finding on the cross-replica re-sync: skipping promotion left a newer
rebuild's signature trusted even after this dispatcher had overwritten its rows
with an older notification. On a warmed Pod A, an own PATCH of f2 commits r+1
and pauses after releasing the user lock, before its adapter upsert; Pod B
deletes f2 at r+2; a concurrent search on A detects the peer change, rebuilds
the scope without f2 and publishes r+2; the pending r+1 upsert then reinserts
the deleted fact, and the revision check merely skipped because the synced
revision was r+2 rather than the PATCH's previous r. The signature stayed equal
to the live manifest, so searches returned the deleted fact indefinitely. The
generation fence could not help: it only suppresses promotion, the stale
adapter mutation still happens.
Rule: after an own commit's notifications are applied, the mutated scope's
recorded signature may stay only when the delta is provably compatible with
the published snapshot -- the scope is synced at exactly previous_revision and
no rebuild published rows for it (or storage-wide) since the fence snapshot --
in which case it is promoted as before. In every other case where this
dispatcher mutated the adapter for the scope (synced at another revision,
newer or older; generation moved; not synced at all; or notifications without
a committed revision, i.e. migration and clear paths) the scope's signature is
forgotten so the next search's compare rebuilds it. The user's other scopes
were not mutated by this delta: they are promoted when provable and otherwise
left untouched. The decision runs under _cache_lock together with the
generation reads, so a rebuild publishing concurrently is either observed or
observes the forget; the failure path (dirty + forget) is unchanged.
Orders covered: the reviewer's (publish r+2, then the stale upsert) forgets and
the next search rebuilds without f2; the earlier fence order (publish between
adapter op and promotion) now forgets too, one redundant rebuild; no concurrent
rebuild promotes exactly as before.
Test: test_stale_own_delta_replayed_over_a_newer_snapshot_forgets_the_scope
holds the PATCH's adapter upsert after the user lock is released (new
pause_next_upsert hook on the test adapter), Event-ordered, red before the fix.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(browserless): send the web_fetch token as a query parameter
fetch_html_with_status put the configured token in the /content JSON body.
Browserless reads the token only from the token query parameter or the
Authorization header, and checks it before reading the body, so with
BROWSERLESS_TOKEN set every web_fetch failed with HTTP 401; against an
instance started without TOKEN the unknown body key failed schema
validation with HTTP 400. capture_screenshot already sent ?token=, so
web_capture worked with the same token.
Both endpoints now take the token from BrowserlessClient._auth_params().
The test that pinned the body token is rewritten, and a stand-in for the
Browserless request gate (query/header auth, no extra body keys) drives
both endpoints through the real httpx request path.
* docs(changelog): reference #6484
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
