* feat(search): find conversations and jump to matching messages
* test(extensions): align example registry with search contribution
* test(search): scope bookmark search to its extension
* fix(search): address review correctness and scaling gaps
* feat(memory): support category capacity reservation in DeerMem hybrid-v1 eviction
In long-running personal assistant interactions, core user profile facts
(such as health allergies, coding style, or identity constraints) can suffer
from capacity starvation as high-confidence conversational context accumulates.
This change introduces opt-in category capacity reservation for DeerMem:
- In DeerMemConfig and select_facts_for_capacity, support eviction_reserved_categories
(defaulting to ['correction'] for historical compatibility) and
eviction_decay_exempt_categories (defaulting to empty list).
- Wire MemoryUpdater to pass the configured categories to select_facts_for_capacity.
- Add deterministic offline unit tests in test_memory_personal_assistant_protection.py
verifying root-cause reproduction, bidirectional lifecycle (clean explicit removal
and contradiction replacement), and strict backward compatibility under defaults.
* fix(memory): preserve correction priority in reserved capacity
Co-authored-by: Wistom Zhang <77424093+Yakyou7@users.noreply.github.com>
---------
Co-authored-by: Totoro <msy626836554@gmail.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* feat(opensandbox): add opt-in secure_access sandbox option
Forward a new `sandbox.secure_access` option (default false) to
SandboxSync.create so OpenSandbox can require a per-sandbox endpoint
token at the ingress gateway and server proxy. The server returns the
token as an endpoint header and SDK 0.1.15 already attaches endpoint
headers to execd and file requests, so DeerFlow needs no token handling.
Non-boolean values are rejected at provider startup so a quoted "false"
cannot silently enable it. Documented in config.example.yaml,
CONFIGURATION.md, the SandboxConfig docstring and the provider README.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(opensandbox): validate server proxy boolean
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Totoro <msy626836554@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
In long-running agent workflows with repeated context compaction, empirical gateway ablations across 132 real sessions revealed that standard boundary compliance remains robust, while semantic drift can occur when model-generated summaries rephrase literal approval phrases and action conditions.
This change converges PR #8114 into a clean built-in summary prompt enhancement:
- Replaces regex extraction, tag elevation, and heuristic revocation with an extended built-in DeerFlow summary prompt (_DEERFLOW_SUMMARY_PROMPT).
- Instructs the summarizer to faithfully record active user requirements, literal approval phrases, permitted actions, and later explicit revisions without paraphrasing.
- Treats summary text strictly as historical data, fully preserving the security boundary and control/data plane separation of DurableContextMiddleware.
- Unifies automatic and manual compaction factory paths to select the built-in template when unconfigured, while preserving explicit summary_prompt precedence.
- Adds comprehensive sync/async unit tests covering built-in and custom templates.
Co-authored-by: Totoro <Totoro-qaq@users.noreply.github.com>
Co-authored-by: Totoro <msy626836554@gmail.com>
* docs: turn README into an index and move reference detail to docs/guide
The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md.
* docs: send user docs to topic guides and budget the root README
The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003).
* chore: remove unreferenced screenshots and finished plan notes
pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them.
* docs: route always-loaded AGENTS.md files to scoped guides
* docs(agents): cut scoped guides to scope, verify, traps, decisions
Each AGENTS.md keeps four kinds of content: the scope of the directory, the
verification commands, traps a local test run does not show, and decisions the
code cannot state. Values, field lists, function inventories, test names, issue
numbers and history are deleted, because the code, the tests and git state them.
Search and fetch providers share one community guide. Four guides with no
remaining content are removed. The size check now allows 6 KiB per file (hard
limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB).
* style: format frontend guide table and guidance test imports
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(agent): ground runtime decisions in structured evidence
Use producer status and structured errors for tool recovery, preserve memory context without lexical confirmation, and remove predetermined receipt and skill conclusions.
Require explicit MCP discovery and authenticated schedule trial actions. Preserve authorization, ownership, isolation, and bounded execution; update documentation and regression coverage.
* test(sandbox): expect read_file initialization error callback
* fix(agent): address evidence review follow-ups
* fix(agent): preserve retry budgets for OpenAI transport errors
* fix(agent): expose tool execution status in model requests
Project validated execution metadata and exact tool text into request-only JSON so status survives provider serialization. Preserve stored results and receipt hashes, cover SDK and graph boundaries, and migrate replay input hashes without changing recorded responses.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(subagents): add thinking_enabled and reasoning_effort config for sub-agents (#4875)
(cherry picked from commit b1083728b23504cb0f1d1a463f8db70c5ec1c5a7)
* fix: resolve CI failures for subagent thinking config PR
- Fix UnboundLocalError in task_tool.py: initialize parent_thinking_enabled
and parent_reasoning_effort before the if runtime block
- Remove unused imports from test_subagent_thinking_config.py
(SimpleNamespace, pytest, resolve_subagent_model_name)
- Fix import sorting in subagents_config.py and test file
- Add reasoning_effort: None to executor test assertion
(cherry picked from commit 187851e307b871937e48bb67ae1b34b8b69bf92d)
* fix: address review comments - remove parent inheritance, supports_thinking clamp, Literal type
- Remove parent_thinking_enabled/parent_reasoning_effort from metadata
extraction in task_tool.py (backward compatible: subagents only think
when explicitly configured)
- Add supports_thinking clamp in _create_agent to prevent cross-model
mismatch errors
- Change SubagentConfig.reasoning_effort to Literal type
- Fix sandbox_state assignment that was accidentally removed
- Fix syntax errors in subagents_config.py (missing closing parens)
- Update all docstrings/comments from "inherit" to "no override"
- Simplify test helper and remove parent inheritance tests
(cherry picked from commit 74f9c659ef4a98bd1756109929ab0748fab82317)
* feat(subagents): adapt reasoning settings to current native workers
* fix(subagents): retain model context on reasoning rejection
* fix(subagents): preserve model-factory reasoning diagnostics
---------
Co-authored-by: Neon549 <neon549@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(wizard): replace retired Claude Sonnet 4 default with claude-sonnet-5-5
The setup wizard's Anthropic default model is now `claude-sonnet-5-5` instead of the retired `claude-sonnet-4-20250514`. A new per-model `model_omitted_config` field makes the wizard leave out the extended-thinking (`budget_tokens`) and thinking-disabled templates for that model only, since Sonnet 5.5 rejects them; the other Claude models are unchanged. The Anthropic examples in `config.example.yaml` and `configuration.mdx` (which used `claude-3-5-sonnet-20241022`) now use `claude-sonnet-5-5` without those settings, and the change also adds a regression test and a CHANGELOG entry.
Notice anthropic-retired-models-2026-10-27 about api.anthropic.com's API describes the change. It was read from a file and is not signed by the provider.
* Update CHANGELOG with PR reference #7585
Added a reference to pull request #7585 in the CHANGELOG.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep IM binding codes on screen and report goal state in replies
Connecting Slack, Discord, Feishu, DingTalk, WeCom, Buzz or QQ now opens a
dialog that keeps the /connect command on screen with a copy button, its
expiry time and the binding status, instead of a toast that disappears.
The bots' /connect replies and the IM /goal replies now use the owner's
web UI language (else channel_connections.notification_locale), and /goal
reports the goal's state as the web goal bar does: in progress, why it
stopped, or that it was met.
* docs(changelog): note the IM binding dialog and localized IM replies (#7591)
* test(channels): check IM reply tables against every notification locale
The connect-reply check compared a fixed en-US/zh-CN pair, and the /goal
reply table had none. Both now match NOTIFICATION_LOCALES with the same
keys in each, like the existing _GOAL_REASON_TEXT check, so a new locale
fails here instead of raising KeyError at reply time.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(streaming): make Redis stream END publication idempotent
* fix(streaming): align Redis test fake with Lua END checks
Remove the unreachable owned missing-stream branch, mirror tail-only data checks in both fake script paths, and pin legacy END and owned missing-buffer behavior against real Redis.
* fix(streaming): gate legacy END scans after writer rollout
* fix(helm): synchronize embedded config version with schema
* test(runtime): use real stream bridge config in close coverage
Keep new configuration defaults in lifecycle fixtures and surface owner-task failures instead of masking them as event timeouts. Drain every helper task and cover context entry/close failures.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): honor the configured recursion limit in IM runs
Every IM channel run asked the Gateway for recursion_limit 100, because
the channel manager seeded each run config with it. The Gateway honors a
request value, so the top-level recursion_limit never reached IM runs,
and a budget of 100 could stop a run after a handful of tool-calling
turns (six in one measurement with the default lead-agent middleware;
the count changes with the middleware chain).
IM runs now send no recursion_limit of their own and get the Gateway's
configured default, clamped by max_recursion_limit, like Gateway API
runs. A session config value still takes precedence, and GitHub channel
runs keep their floor of 250. The README and config.example.yaml IM
examples no longer pin session.config.recursion_limit to 100.
* docs(changelog): reference #7521 for the IM recursion-limit fix
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(projects): search shelf documents by keyword
Add search_project_documents: two-tier name/summary then bounded content scan, case-insensitive substring matching with original-text offsets. Never converts on the fly; reports stop_reason, content_scanned, skipped reasons, and capped unscanned/truncated ids.
* fix(projects): derive unscanned count without paging after budget trip
Review feedback: once the content budget trips, stop paging list_active to enumerate unscanned ids — cap ids at the trip page and derive the honest total from one count_active query (tier 1 is exhaustive, so active - tier1 hits - processed = unscanned). Also add a tool-level isolation test pinning that search never crosses projects or users.
* fix(projects): scan the shelf over one stable snapshot, not offset pages
Review feedback: offset paging re-read the live shelf per page, so a concurrent upload/trash between pages could duplicate a match or silently skip one while still claiming complete coverage. Tier 1 now materializes the whole shelf in a single SELECT (one read snapshot) and tier 2 iterates those same rows; the budget-trip unscanned count is derived exactly from the snapshot with no extra query. Mid-scan upload/trash regression tests pin both failure modes.
The two-Gateway harness checklist in CONFIGURATION.md said a crashed
leader hands its providers over within lease_seconds + grace_seconds; a
peer only tries to acquire on its fixed lease_seconds // 3 tick, so the
worst case is one renewal interval more (50 s with the defaults), as the
leader-lease section of the same file already says. The grace_seconds
field description had the same shorter bound.
The channels.leader_lease comment in config.example.yaml still said web
UI credential changes reach the holder only when it next starts the
channel; since #6988 the holder applies them within one renewal interval.
The harness `check` footer still listed the IM channel gates as waiting
for leader election, which landed in #6843; it now points at the manual
channel item in CONFIGURATION.md, which also gains the runtime-config step.
Exa web_search and web_fetch accept a per-tool base_url, falling back to
EXA_BASE_URL and then the SDK default. Overrides are validated like the
Serper base URL. web_fetch screens a configured backend with the shared
delegated-backend guard.
Co-authored-by: lachy3433 <lachlanallen.co@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(community): add Jsonify dataset rows tool
Adds deerflow.community.jsonify.tools:dataset_rows_tool, a read-only tool
that returns rows from maintained Jsonify datasets. Jsonify pipelines run on
a schedule and repair themselves when a source changes, so the agent reads a
current table instead of scraping.
- operator configures a named map of dataset -> serve URL; the model picks
by name and never sees the URL
- serve URLs must be https without userinfo and pass validate_public_http_url
before any request; the URL is redacted from error text
- paging with has_more/next_offset, version pinned after the first page,
limit clamped to the configured max_rows, system columns stripped
- config.example.yaml entry under tools; unit tests cover paging, clamping,
unknown dataset, unconfigured tool, http/userinfo rejection, private
address block and error redaction
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrmeK98ZKoxZNiUiz8kVeX
* fix(community): harden Jsonify tool paging, logging and error handling
- Stop paging on an empty page or a next_offset that does not advance
- Log config validation failures by field location and error type only
- Validate page shape; malformed or non-JSON pages return a clean error
- Transport errors report the exception type, never the URL
- Add truncated flag when rows were clipped or more rows remain
- Tests for each case
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrmeK98ZKoxZNiUiz8kVeX
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* test: stabilize reporter evidence and guard missing chat nodes
* feat(models): scope Web model choices to tasks, turns and conversations
* test(e2e): mock model-control preview for ordinary chat submissions
* fix(models): harden scoped choices and stale Agent submissions
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* feat(decisions): add an OpenAI Decisions backend for the decision slots
Issue #7005 asked for a second decision provider behind the interfaces the
TypeSafe (Jev) integration already uses, not a new decision system.
`deerflow/openai_decisions` speaks `POST /v1/decisions`: a `noul` question
becomes a `predicate` whose true/false rubric folds into `instructions` (the
endpoint has no criteria field, and dropping it would move the decision
boundary while the adapter kept reporting the same criteria in its policy
identity), answers arrive as an array rather than a mapping, and a refusal is an
answer `type` rather than an HTTP failure -- so it lands in the existing
per-question error channel and each consumer keeps its own failure direction.
Transport is reused rather than copied. `provider_label` and the connection's
`endpoint_path` are the only two seams a dialect needs, so retry and backoff,
the deadline budget, the bounded never-decoded body read and the error taxonomy
keep one owner; `resolve_decisions_connection` pairs the two so a Decisions
connection cannot be built pointing at System One. The three adapters override
their connection resolution, their client, and the value the endpoint judges
(`input` text instead of a `state` object), and inherit thresholds, rubrics,
caches and audit policy unchanged. Wiring stays a `use:` class path, so no
config schema is added.
* fix(decisions): isolate endpoint identities and correct diagnostics
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): arbitrate IM transports with a per-provider leader lease
Every Gateway instance connected every enabled IM platform with the same
bot credentials, so channels could not run on more than one instance
(Telegram 409 storms, Discord answering twice, scattered per-process
state). channels.leader_lease was reserved by #6746; this implements it.
- channel_leases table (migration 0041): one row per provider. Acquire is
one dialect-native upsert decided against the database clock and bumps
a fencing epoch; renew extends only a live lease at the holder's epoch;
release sets the expiry to 0 so a peer takes over on its next tick.
- app/channels/leadership.py: the holder's local monotonic deadline
(send time of the last confirmed renewal + lease_seconds); holds() turns
false one renewal interval early, so the transport stops at least
grace_seconds before a peer may connect. DB calls are bounded.
- ChannelService: start fence in _start_channel (no DB call) behind every
start path; fixed-rate supervisor renews/acquires and runs transport
starts/stops as per-provider tasks; takeover reloads the provider's
saved config; failed starts back off; shutdown releases only leases
whose transport stopped. get_status reports leader/leader_here and the
holder's transport state on every instance.
- Notification worker leaves rows of providers another instance holds.
- Gate: the enabled=false refusal now recommends auto/true; an explicit
true with database.backend memory is refused when a channel is enabled.
- Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated.
* docs(changelog): link the channel leader lease entry to #6843
* fix(channels): see peer-held leases for providers absent from local config
Review on #6843: a provider connected from the web UI on a peer never
enters this instance's channel config, so the notification worker here
still claimed its rows (and parked them for 900s) and GET /api/channels/
reported no leader for it. Every supervision round, and startup, now also
reads the whole lease table once (bounded), and notification exclusions
and status use those observations: a provider whose live lease a peer
holds is left to it even without local config.
Running that read concurrently with renewals exposed that, through
SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on
another connection can hand back one of its own bind values instead of
the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do
not show it). A lease must never take such a row for a confirmed
renewal, so acquire/renew/release now decide by affected-row count and
acquire reads its epoch back in the same transaction.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): parse Slack and DingTalk allowed_users fail-closed
Slack and DingTalk still built their allowlists with their own
_normalize_allowed_users: every entry went through str() without
stripping, only list/tuple/set counted as collections, and the gate was
`if self._allowed_users and ...`. A configured list whose entries are all
blank, such as `allowed_users: [$SLACK_USER_ID]` with the variable set to
an empty string, became an empty set and let every sender through.
" U123 " never matched U123, and a frozenset was stringified into one ID.
Parse both with the shared parse_allowed_users helper that Telegram,
WeChat and QQ use, and gate on `is not None`: unset, null, empty
collections and a blank string allow everyone; a scalar is one entry;
strings are stripped and integers keep matching their string form, as
before; booleans, floats, None and mappings are rejected (a mapping as a
whole); a configured value with no valid ID denies every sender and logs
an error. A scalar containing commas or whitespace warns that it stays
one literal ID.
* refactor(channels): share the Slack and DingTalk allowlist coercer and separator warning
Slack and DingTalk had byte-identical ID coercers and their own copies of
the scalar-separator warning. Move the coercer into allowed_users.py as
parse_string_or_int_user_id and opt both channels into
warn_on_scalar_separators, as QQ and WeChat do. Log messages are unchanged.
Pin that a separator-only string such as "," stays one literal ID
(denying everyone else) and still warns.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(buzz): treat a scalar allowed_users entry as one member pubkey
BuzzChannel built its allowlist by iterating config.get("allowed_users") directly. A scalar string -- what an allowed_users: $BUZZ_OWNER_PUBKEY reference resolves to -- was iterated character by character, feeding each character to parse_pubkey, whose first non-hex character raised and aborted channel construction; a numeric scalar raised TypeError the same way. The sibling QQ (#6730) and Discord (#6776) connectors already treat a scalar as a single entry.
Parse the value with the shared parse_allowed_users helper via a parse_pubkey coercer: a scalar is one entry, list entries keep their per-entry validation, unusable entries (bad hex, numeric refs, mappings) are dropped with the shared warning instead of crashing, and the deny-by-default contract is unchanged -- an empty or blank value still yields an empty allowlist with start()'s existing deny-all warning.
config.example.yaml states the accepted shape next to the key, matching the mention_free_channels note added by #6776.
* docs(config): split buzz allowed_users note into aligned comment lines
The two-line note on the buzz example entry was emitted with a literal
`/n` where a newline was intended, so the second note never became its
own comment line and the `#`-column alignment shared with the
DENY-BY-DEFAULT lines below was broken. Split it to match the
mention_free_channels sibling shape (review comment on #6811).
* fix(buzz): accept a single mention-free channel id
`_mention_free` iterated the configured value directly, so one unquoted
UUID became a set of its characters and a numeric or boolean scalar
raised TypeError while the channel was being constructed. Both shapes are
documented as valid for the sibling Discord connector.
Parse the key through a dedicated helper that follows the Discord
convention: a scalar is a single entry, list entries are kept verbatim,
and an unusable entry never matches so `require_mention` stays in force.
* fix(channels): share scalar-or-list channel ID parsing
* docs(channels): keep guidance within CI size budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): add channels.leader_lease and gate it under multi-process signals
IM channels other than GitHub connect outbound to their platform. With more
than one Gateway instance each one opens that connection with the same bot
credentials: Telegram rejects the second poller with 409, Discord answers every
message twice and per-process ChannelManager state diverges. The multi-process
startup gate said nothing about channels; the only guard was a comment in the
Helm values.
Add the `channels.leader_lease` section (`enabled: auto|true|false`,
`lease_seconds`, `grace_seconds`; startup-only, registered in the reload
boundary) that the per-provider leader lease will read, validated by a Pydantic
model from the non-schema `channels` mapping. The gate now refuses
`enabled: false` under a multi-process signal while a non-GitHub provider is
enabled under `channels.*` or connectable through `channel_connections`, naming
the providers and the rollback; a malformed block is refused on every topology.
`auto` follows the multi-process signal, so a single instance is unchanged.
ChannelService strips the block before its per-provider loops and exposes it
as `leader_lease`; `ensure_ready_channels` only visits registered providers.
The two-Gateway harness overlay leaves the block alone. config_version 60 in
the example, the Helm chart and its README; docs replace the comment-only
"keep 1 replica" guidance with the gate.
* docs(changelog): link #6746
* fix(channels): stop recommending the unimplemented leader lease as a remedy
Review on #6746: the refusal told operators to switch channels.leader_lease
from false to auto/true "so one instance owns each provider's connection", but
the transport arbitration ships separately, so that change only bypassed the
clause and every worker still started the same bot.
The refusal now offers only the remedies that work today (a single instance, or
disabling the channels) and states that auto/true are reserved for the lease
that ships separately. A regression helper in the gate tests asserts the
remedies start with the rollback and that auto/true are never offered. The
config example, Helm values/README, CONFIGURATION.md, README, the deployment
guide (en/zh), the model docstrings and both changelogs describe auto/true as
reserved settings rather than an ownership guarantee.
* fix(channels): warn on an enabled unknown channel key; reword the changelog entry
Review on #6746: the registry guard in ensure_ready_channels skipped a
misspelled provider key (channels.telegam) silently, where it previously
reached _start_channel and logged 'Unknown channel type'. An enabled dict
block that is neither a registered provider nor leader_lease now logs a
warning that lists the registry.
The changelog entry claimed the chart and docs drop the "keep 1 replica"
guidance; they keep it (auto/true do not make several replicas safe yet) and
only stop relying on the comment alone.
* fix(dev): disable UI channel providers in multi-instance overlay
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(qq): parse allowed_users with the shared fail-closed allowlist
QQChannel built its allowlist with set(config.get("allowed_users") or []).
A single OpenID, which is what an `allowed_users: $QQ_ALLOWED_USERS`
reference resolves to, became a set of its characters, so the configured
user was silently dropped. A mapping authorized every key, including one
mapped to false, and a numeric scalar raised TypeError while the channel
was being constructed.
Parse it with the shared parse_allowed_users helper that Telegram and
WeChat already use: unset, null, empty collections and a blank string
allow everyone; a scalar OpenID is one entry; entries that are not
non-blank strings (such as numeric QQ account numbers) are dropped with a
warning, and a mapping is rejected as a whole; a configured value with no
valid OpenID denies every sender and logs an error. A scalar containing commas or whitespace warns
that it stays one literal ID. Binding codes still precede the allowlist.
* test(qq): gate group messages by member OpenID in allowlist tests
Every allowlist test sent a C2C frame, and the shared event() helper puts
the sender in both user_openid and member_openid, so a group-path
regression in sender resolution would pass unnoticed. admits() now takes
a kind; group frames carry only member_openid and a group_openid that
differs from the sender. The listed-OpenID test runs for both kinds, and a
new test shows that listing the group's OpenID does not admit its members.
Document that a blank or whitespace-only allowed_users string allows every
sender, like an unset value or an empty list.
* refactor(channels): share scalar allowlist separator warnings
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): render Feishu option clarifications as button cards
When the agent's ask_clarification carries options, the Feishu channel
now renders the question as an interactive card with one button per
option instead of plain text. This is the outbound half: card
construction plus a process-local pending registry that a later
card-action callback handler will validate clicks against.
- New feishu_card_interaction module owns request extraction, button
card rendering, and the pending registry (30-minute TTL, matching the
typed-reply clarification window)
- Manager attaches the clarification's structured human_input payload to
the final outbound's metadata so channels can render richer
affordances without touching the Gateway protocol
- Feishu channel renders the button card only when
card_interaction_enabled is set (default off); free-text, form, and
over-four-option clarifications keep the plain-text path
- Button cards bypass the markdown wrapper via a raw_card_content flag
on the card send/patch helpers; plain-text call signatures unchanged
* feat(channels): close the Feishu button-card click loop
The inbound half of interactive clarification cards: a click on an
option button validates, resolves the clarification, and marks the
card in place.
- Register the card-action callback only when card_interaction_enabled
is set; clicks arrive as WS EVENT frames through the SDK's callback
processor map, and the synchronous return value becomes the toast ack
- consume_if_originator atomically validates operator identity against
the pending entry: non-originators get a refusal toast and the entry
survives; unknown or expired requests are toast-only and dropped
- A validated click consumes once, updates the card to a terminal
resolved state, and injects a hidden inbound carrying the
human_input_response so the run resumes on the originating thread
with Web-card parity
- Manager translates the inbound's human_input_response metadata into
the human message's additional_kwargs with hide_from_ui set
* docs(channels): document Feishu interactive clarification cards
- README / README_zh: optional card_interaction_enabled setup, calling
out that card.action.trigger lives under the console's callback
configuration tab (separate from event subscriptions) and requires
publishing a new app version
- config.example.yaml: commented flag with the same console caveat
- backend/AGENTS.md: module ownership note for feishu_card_interaction
* fix(channels): close Feishu card-click routing and ack gaps from review
Address review on #6670:
- Carry the routing identity (topic_id, connection_id, owner_user_id)
on the pending entry and the synthetic inbound, so clicks resolve to
the clarified thread in topic-scoped chats and pass bound-identity
admission instead of opening a new thread or being rejected
- Validate option_value against the registered options at consume time:
an unknown value is treated like a stale card and the entry is kept
- Return a warning toast when a click cannot be dispatched, and flip
the card to its resolved state only after the inbound is durably in
the bus — never display a choice that was not recorded
- Drop the dead mark_sent/consume registry APIs and pass the payload's
own option ids through to the buttons
- Add an end-to-end test through the manager's thread resolution
pinning that a card answer reuses the clarified thread
* fix(channels): carry workspace_id on the Feishu card-click inbound
The connections-enabled admission path re-reads the binding with the
message's workspace_id, and Feishu connections are bound with
workspace_id = chat_id. The synthetic inbound set connection_id and
owner_user_id but never workspace_id, so the re-read missed and every
click was rejected as unbound after the entry was consumed.
- Set inbound.workspace_id from the entry's chat_id (Feishu's
chat-as-workspace binding convention)
- Extend the card-click e2e to require_bound_identity=True, driven
through the channel's real _resolve_card_click production
* fix(channels): stabilize Feishu clarification delivery and routing
Option cards could fail through the real reply SDK, resume another conversation,
or leave a stale text hint after the answer was accepted.
Unify raw-card delivery and persist trusted thread aliases before showing buttons.
Keep protocol request IDs separate from bounded, one-time callback tokens; validate
originators and option pairs, and retire only the accepted card's text/button state.
Treat valid hidden human answers as turn boundaries so old replies and artifacts
do not leak into the next answer. Cover consecutive clarifications and /new routing
through real SDK request models, stores, and the run journal.
Validation: 147 focused tests and 29,421 offline tests passed (405 skipped);
strict blocking-I/O passed 245 tests (1 skipped). Ruff and guidance checks passed.
* fix(channels): preserve clarification context in Feishu cards
Button cards replaced the full clarification text but omitted its background,
leaving users without the context needed to choose an option.
Carry optional context from request metadata and render its original Markdown
before the question. Keep question-only cards for absent, blank, or invalid context.
Cover rendering and all real SDK delivery paths, and update both user guides.
Validation: 156 focused tests, 245 strict blocking-I/O tests, Ruff, and guidance
checks passed. The offline run recorded 29,381 passed, 49 failed, and 405 skipped;
48 import failures reproduced before the fix. All 49 cases passed on both versions
in isolated environments, and the Feishu-to-export order check passed 13 tests.
* feat(community): add You.com web search provider
Adds You.com as an opt-in community `web_search` engine, mirroring the
existing community engines (tavily, brave, exa, firecrawl, groundroute).
You.com runs its own web index and returns extracted page snippets per
hit rather than a single meta description, so a research loop gets usable
context without a follow-up fetch.
The provider picks its endpoint by whether a key is configured, so it
works out of the box and scales up when the operator adds one:
- no key -> /v1/agents/search, keyless free tier
- YDC_API_KEY -> /v1/search with X-API-Key, full Search API
Endpoint and headers are always chosen together; the keyless endpoint
rejects an auth header, so a stale key can never be sent there. Both
return the same body shape, so one normalizer covers both.
httpx only, no new dependency and no pyproject extra, so uv.lock is
untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint
* fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY
The doctor detail returned before any key check, so operators who had
set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'.
Reworded to match the sibling free-provider entries and stay accurate in
both states.
* fix(search): sweep youcom max_results onto the shared owner
The single-owner gate that landed with #6563 flags this branch's private
_coerce_max_results as a newly hand-rolled copy, so backend-unit-tests
now fails after merging main. Follow the coordination note from review:
delegate to deerflow.community.search_max_results.coerce_max_results,
which owns the rejection profile (booleans, non-integral floats,
OverflowError), and keep only the 1-100 mirror of You.com's server-side
count clamp on top — clamping policy stays per-provider.
The .inf regression test now goes through the public tool path and also
pins the bool and non-integral-float shapes of the shared bar.
* docs(tools): use the bare you.com platform link in config and docs
Match every other provider entry: no other provider in config.example.yaml
or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params
from the three you.com/platform links. Integration attribution stays in
the identifying User-Agent sent to You.com hosts.
* test(youcom): cover shared max_results ownership and bounds
Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters.
---------
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* Fail closed on delegated fetch backends to close SSRF gap
* fix(community): resolve delegated-fetch credentials from one config snapshot
Address review feedback on the fail-closed SSRF guard:
- firecrawl/fastcrw client construction no longer re-reads get_app_config(),
so the API key and endpoint come from the same revision the endpoint screen
validated (prevents cross-revision credential misrouting).
- fix the browserless test double to accept the new two-argument
_get_browserless_client signature.
- document the check-only (DNS rebinding) caveat on
validate_delegated_backend_url.
- add a CHANGELOG upgrade note for the default self-hosted backend break.
* fix(doctor): consolidate delegated fetch backend checks and deployment guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Opt-in per-document summary column rendered into the injected
<documents> index, list_project_documents JSON, and the documents API.
Pipeline (projects/summaries.py, behind the narrow enqueue_summary
interface):
- The two shelf write routes enqueue a job on created=True; dedup hits
never regenerate. Off by default (projects.summaries_enabled).
- Bounded worker pool (summary_concurrency + bounded summary_queue_size;
a full queue drops the job) fed by FIFO; shutdown drains queued jobs,
delivers one sentinel per worker via blocking put (queue smaller than
the worker pool cannot strand idle workers), and bounds only
genuinely in-flight jobs with the grace period.
- Jobs carry (document_id, project_id, user_id) explicitly; the job's
owner is bound into the user ContextVar around the model call so
tracing/usage attribution never falls back to 'default'.
- Text source reuses read_text_serving_path (convertible-first rule,
content validation); conversion has no deadline (await_drained), the
model call has a caller-side wait limit — the worker slot is retained
until the provider call actually finishes and late results are
discarded.
- Output contract: strip leading <think> reasoning -> unwrap fence/
quotes -> collapse to one line -> empty is NULL -> UTF-8 byte cap on
a character boundary; length-capped responses (provider-normalized
detectors) are failures.
- set_summary pins updated_at (literal self-assignment) so a landing
summary never reorders the shelf; owner-predicated UPDATE is a no-op
on rows purged mid-generation.
- Metrics: eligible/success/failed/skipped/queue_dropped/
uncapped_accepted.
Schema/config:
- project_documents.summary (migration 0036, nullable, no backfill;
forward-compat floor unchanged).
- Six projects.summary_* keys, all registered in STARTUP_ONLY_FIELDS
(captured once at lifespan startup); config_version 57 -> 58.
- run_oneshot_llm gains an optional model_overrides passthrough (passed
only when set, existing callers unchanged) and a detailed variant
exposing the raw response message for termination-signal detection.
Docs: API.md response field, ARCHITECTURE.md pipeline paragraph,
migrations AGENTS.md chain head, root AGENTS.md records the scoped
all-core exception with the lift-out path.
* feat(goal): record the met goal and keep goal state on the history head
- Add a goal_outcome LastValue channel (GoalOutcomeState). Only the
guarded clear of a satisfied ordinary goal writes it, in the same
checkpoint as the clear; the reply anchor is the latest visible AI
message with text, the reply the visible signature keys on. A goal
stored without created_at still clears and gets a record.
- write_thread_goal removes the record on every other goal write and
lists it in channel_versions, new_versions and metadata writes when
it changes. Threads that never had a record keep their checkpoint
shape. The sync-only SqliteSaver and CachedHistorySaver, which take
the #6448 sync fallback, store and drop it the same way.
- goal_outcome is server-owned: POST /state and non-internal run input
that carry it get 400. A POST /state goal write, or a goal in
external run input, clears it.
- A branch head written onto its replay base resets the channels the
head lacks, so a goal or record that write_thread_goal removed does
not come back. Branching from the met turn keeps the record without
the met goal.
- The POST /history head now carries an active goal and an achieved
goal_outcome, also before any message exists, so the post-run
history refetch keeps them. Older entries stay title/thread_data.
- contracts/thread_goal_contract.json pins the chat stand-down and
check-failure codes, the record shape and status, the head keys and
the host-written reasons.
* feat(frontend): show why a goal stopped and when it was met
- The goal bar derives its state from the active goal, the goal_outcome
record and the messages: set, in progress, continuing, stopped (limit,
auto-continue off, no progress, token budget, missing evidence, failed
run, unknown code), waiting for you, waiting, couldn't check the goal,
paused and goal met. An 8/8 stand-down now reads "Stopped" with the
next step instead of "Continuing 8/8", and a count-0 stand-down no
longer looks like a fresh goal.
- "Goal met" shows while its anchor reply is the latest visible message
(a visible human message, or a visible AI message with text, moves the
chat on), so a reload agrees with the live view. An active goal wins
over the record, and this tab's /goal set or clear hides it.
- useActiveGoal drops the optimistic goal when the record of that same
goal arrives, which fixes the bar that stayed after a goal met on the
first run (the clear frame has no goal key), also for a goal stored
without created_at. Server goal values pass through isGoalState; a
/goal status read uses the backend's own active check, so the bar, the
toast and the edit lock agree with the server.
- Row 2 gives the reason and what to do next; Details holds the full
goal, the checker's own note (never host-written text, a check failure
or a paused goal's older check), the assumption note and an unknown
code, and starts collapsed in each new state. The bar is a labelled
section with a live region that announces state changes only (another
goal's same state counts) and re-reads its text after a language switch.
- The edit pencil on the latest turn is shown disabled with "Editing is
off while a goal is set" instead of hidden; the message toolbar shows
on focus-within, so keyboard users can reach it.
- A /goal set or clear rejected with 409 keeps the draft and says a run
is still going; other failures keep the server's message and the draft.
A /goal request replaced by a newer one leaves the composer alone, so
a double submit no longer empties it before the second one fails.
- Copy lives in inputBox.goalBar (en-US, zh-CN); a contract test reads
contracts/thread_goal_contract.json, and the raw-identifier guard now
catches stand-down codes and goal field names.
* test(frontend): cover the goal bar on both chat pages
- tests/e2e/goal-status.spec.ts runs every case on the chat page and the
agent chat page, since each wires the goal bar and the edit lock itself:
a goal stopped at 8/8, one waiting for details and one waiting on an
open question card show their reason and next step, also after a
reload; the locked edit pencil gets keyboard focus, shows its toolbar
and tooltip, does not open the editor, and hides while a file uploads;
Goal met after one automatic follow-up (In progress, Continuing 1/8,
then "Auto-continued once") survives a reload and hides after the next
visible message; a goal met on the first run of a new chat replaces the
set bar and frees the pencil, and a new /goal replaces Goal met while
its own run goes on; a /goal set or clear refused with 409 keeps the
draft for the retry and shows its own busy toast.
- The mock POST /history head carries only the contract's
history_head_keys, with goal only while active and goal_outcome only
when achieved; thread state and run values omit an unset goal instead
of sending goal: null. PUT and DELETE /goal drop the met record,
setGoalWritesBusy returns the Gateway's 409, and a repeat create of a
requested thread id returns the thread as is, like the Gateway.
- The raw-identifier guard also catches checker blockers and the met
record's field names.
- frontend/src/AGENTS.md: the GoalStatus sentence now names goal,
goal_outcome and useActiveGoal, reworded in place.
* test(frontend): keep the draft through a failed upload and a busy /goal
#6412 fires onSent only once a send is dispatched, so a failed upload
keeps the text, files and stored draft. A /goal the Gateway refuses with
409 never reaches sendMessage, and the composer rejects it so PromptInput
keeps the text. Drive both through the real sendMessage in one composer:
a failed upload keeps "summarize it" stored, a busy /goal keeps its text
and stored draft with no run, and the retry saves the goal and starts
one run with the objective, not the "/goal" text. After that run the
composer is empty and no stored draft comes back.
* fix(frontend): drop the stop condition's closing period before the pause clause
A stored stop condition often ends with a period, so the schedule card
read "…remain)., pauses itself" (and "…。,满足后自动暂停" in Chinese).
Trim the condition's closing sentence punctuation before the clause that
follows it. The event line, where the condition ends the sentence, is
unchanged.
* fix(frontend): point "evidence missing" at the agent, not a user reply
"Evidence missing" asked the user to reply with what's left, which is
the push a goal is meant to save. It now asks the agent to show the
result or explain what's missing, and offers /goal clear only for a goal that
no longer applies: clearing drops the met record, so work that is right
would never show "Goal met".
zh-CN follows. Copy only: no key, state or mapping changes, and the
scheduled page's reason labels are untouched.
tests/unit/components/workspace/goal-status-next-steps.test.ts checks
the row in en-US and zh-CN, and that no stop row offers /goal clear
without that condition. goal-status.dom.test.tsx and
goal-stream.dom.test.tsx now expect the new copy.
* fix(frontend): say "Couldn't check the goal" once in the goal bar
After a goal check that could not run, the bar's second line borrowed
the scheduled page's whole reason, so it repeated the chip: "Couldn't
check the goal · Couldn't check the goal; the run itself may be fine.
It's checked again after your next message." Chinese had the same
repeat.
The line now gives only why the check failed ("The run itself may be
fine", "No final reply was saved" or "The chat changed during the
check"), then when it is checked again. The screen-reader announcement
follows. The chip and the scheduled page's reason labels are unchanged.
tests/unit/components/workspace/goal-status-unchecked.test.ts checks
every check-failure code in en-US and zh-CN: the label appears once,
and the line still says why the check failed.
* feat(goal): redact the evaluator input when pii_redaction is enabled
The goal evaluator calls its model directly, outside
PiiRedactionMiddleware, so it sent raw user lines, tool arguments, tool
results and Human Input Card answers even with pii_redaction enabled.
With pii_redaction enabled:
- the messages in the evidence window are redacted whole before the
evidence caps can cut an identifier in two, as TitleMiddleware does.
This reuses redact_queued_messages, which already covers content,
call arguments and card answers. Earlier messages are left alone: the
evaluator never sees them, and the pass is synchronous work on the
event loop (0.84 s for an 800-message, 4 MB thread, 0.06 s for its
window);
- the assembled input, the goal objective included, is redacted again.
A redaction error fails the check with evaluator_failed; raw text is
never sent instead. When redaction is off, the input is byte-identical.
Thread state keeps the raw text. README.md, config.example.yaml and the
middleware docstring list the evaluator with the other redacted inputs.
tests/test_goal_pii.py puts an address where the evidence caps cut a
tool argument, a tool result and a card answer, and checks the
objective, the unchanged input when redaction is off, that only the
evidence window reaches the redactor and gives the same evidence as
redacting the whole thread, that a redaction error stands down with
evaluator_failed before the model is called, and the worker path.
* fix(goal): redact the hidden continuation when pii_redaction is enabled
The hidden /goal continuation repeats the goal objective and the
evaluator's reason and evidence summary. PiiRedactionMiddleware skips
it, since it is a framework message and not a user message, so with
pii_redaction enabled each continuation turn sent them raw to the
agent's model, and the message stayed in the thread for later model
calls. The durable-context block and the evaluator input already redact
the objective.
With pii_redaction enabled, make_goal_continuation_message now redacts
the objective, the reason and the evidence summary with redact_text
before the message is built. The thread keeps the redacted message,
which the UI hides; the goal state keeps the raw objective.
The worker builds the message before it counts the continuation. A
redaction error fails the check the way an evaluator redaction error
does: run_failed with stand_down_reason evaluator_failed, the goal stays
active, nothing is sent and the continuation budget is not spent. Only
the exception type is stored. With redaction off, the message is
byte-identical.
README.md, config.example.yaml and the middleware docstring list the
continuation with the other redacted inputs.
tests/test_goal_continuation_pii.py checks the redacted objective,
reason and evidence summary against the tokens redact_text gives, the
unchanged message with redaction off or no config, the worker path
under the run config, and a redaction error.
* docs(changelog): link the goal outcome entries to #6556
Fill the four ([#PR]) placeholders in CHANGELOG.md and CHANGELOG_zh.md with this PR's number and add the matching link references.
* fix(frontend): keep an active goal stored without timestamps visible
POST /threads/{id}/state and run input can store an active goal without
created_at or updated_at. The backend still evaluates it and rejects an
edit with 409 (is_active_goal checks only status "active"), but
isGoalState required both timestamps, so history and live values
dropped it: both chat pages hid the goal bar and unlocked the last-turn
pencil, and only a /goal status read, which used a looser predicate,
brought them back.
isGoalState now accepts any record with a string objective and status
"active", and normalizeGoalStatusRead uses it, so thread values, live
updates and status reads share one rule. Malformed or inactive values
are still dropped.
Tests cover a sparse active goal from the history head and from a live
update before the post-run refetch (bar shown, hasGoal true), and a
malformed goal that stays hidden.
* test(goal): check the outcome write metadata the same way on every saver
PostgresSaver stores checkpoint metadata through
get_serializable_checkpoint_metadata, which drops "writes", so the
[postgres] case of test_goal_outcome_round_trips_through_the_sync_fallback
failed with KeyError: 'writes' when CI ran it against a live database.
The test now records the metadata write_thread_goal passes to put and
asserts the clear's and the next goal's "writes" there, for every saver.
The stored channel values are checked as before, and on sqlite and
cached-sqlite the stored "writes" must still equal what was passed.
* docs(agents): keep goal_outcome out of the agents guidance chain
The middlewares AGENTS.md chain is already over the 98,304-byte AG002
hard limit on main, and the PR-mode check fails any chain a PR touches,
so adding goal_outcome to the ThreadState field list in
deerflow/agents/AGENTS.md failed the agent-guidance job. goal_outcome
stays documented in runtime/AGENTS.md and backend/app/gateway/AGENTS.md,
which are outside that chain.
* refactor(goal): read the evaluator's evidence and redaction from one window
format_visible_conversation recomputed the window start that the redaction
pass takes from _evidence_window. Both now use _evidence_window, so a later
change to the cap or to what counts as visible moves both.
The window test now fills the window with tool-only replies and tool results.
Before, it caught a cap change in one copy but not a change to what counts
as visible.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(auth): key the compose login lockout by client, not by nginx
Failed logins are counted per client IP, and the Gateway honors X-Real-IP
only from a peer listed in AUTH_TRUSTED_PROXIES. The compose files never
set it, and every browser request reaches the Gateway from the nginx
container, so all logins shared nginx's address: five wrong passwords
from anyone locked every user out for five minutes.
AUTH_TRUSTED_PROXIES now also accepts hostnames, resolved off the event
loop and cached for 10 seconds (failed lookups included, and only when
X-Real-IP is present), and both compose files default it to the bundled
nginx service, whose address Docker assigns. nginx overwrites X-Real-IP
with $remote_addr on every Gateway route; a test pins that for the three
maintained configs.
* docs(changelog): reference #6519
* fix(auth): share one trusted-proxy lookup across concurrent cache misses
A burst of setup-status / login requests on an expired hostname entry
started one getaddrinfo per request while DNS was slow, occupying the
default executor that password verification also uses. The cache check
and an in-flight task registration now happen without a yield between
them; later callers join that task through asyncio.shield, so a cancelled
request cannot cancel the lookup the others joined, and the task caches
its own result.
The compose default keeps `:-` on purpose: every browser request arrives
through nginx, so an empty AUTH_TRUSTED_PROXIES would only bring back the
shared lockout. The compose comment and .env.example say so, and the
rendered-compose test now covers the explicitly empty value.
* fix(docker): keep a .env AUTH_TRUSTED_PROXIES under make docker-start
docker.sh start runs Compose from docker/ without --env-file, so the dev
compose file's ${AUTH_TRUSTED_PROXIES:-nginx} interpolated from the shell
only, and the resolved environment entry outranked the operator's value
loaded through env_file. Export it from .env alongside the proxy variables
(a shell export still wins), and strip a CRLF before the quotes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(auth): count login lockouts in the database across Gateway replicas
POST /api/v1/auth/login/local throttled failed logins per client IP in a
module-level dict, so with N Gateway replicas behind one load balancer an
attacker got N x auth.local.max_login_attempts guesses and a lockout on
one replica was invisible to the others.
Move the counter behind a LoginThrottleStore (deerflow.persistence.
login_throttle): MemoryLoginThrottleStore keeps the historical per-process
semantics and 10k tracked-IP bound; SqlLoginThrottleStore shares one
login_throttle row per IP through the application database, incrementing
and starting the lock in a single dialect-native upsert, applying check
decisions with compare-and-set predicates in a separate write
transaction, and sweeping served locks plus 24h-idle counters in bounded
batches. Both stores pin the existing contract: the lock starts when the
count reaches max_login_attempts, the committed duration is honored when
the policy shrinks mid-lock, a served sentence is never resurrected,
tightening keeps the count, success resets the IP, and the 429 status and
message are unchanged.
New startup-only selector auth.local.throttle_storage (auto | memory | db,
default auto = the database whenever database.backend is sqlite or
postgres). The Gateway installs the store once at startup after the
persistence engine; deps._validate_login_throttle_storage warns when a
multi-process deployment keeps memory. Migration 0034_login_throttle
creates the table; config_version 55 -> 56 in config.example.yaml and the
Helm chart. Docs: README, CONFIGURATION.md, AUTH_DESIGN.md, config and
migrations AGENTS.md, CHANGELOG.
* docs(changelog): link the login throttle entry to #6501
* fix(auth): re-check expiry on the row the login-throttle sweep deletes
The bounded cleanup in SqlLoginThrottleStore.record_failure deleted
`WHERE ip IN (SELECT ip ... WHERE <served or stale> LIMIT 200)`. On
PostgreSQL READ COMMITTED the expiry predicate applied only inside the
snapshot-bound subquery: when the DELETE had to wait on a row lock held
by a concurrent check() that was extending the same lock under a raised
lockout_seconds, PostgreSQL re-evaluated only the outer WHERE on the
updated tuple, IP membership still held for the old candidate, and the
now-active lock was deleted; the next attempt from that IP got a clean
record and reached password verification (review of #6501, P2).
Repeat the served/stale predicate on the DELETE target as well
(SqlLoginThrottleStore.sweep_statement), keeping the bounded candidate
subquery, so the re-evaluation sees the updated row's expiry. The other
writes already carry their full compare-and-set predicate on the target
row: _discard and the check() UPDATE both use _matches (ip, fail_count,
locked_at, lock_duration_seconds), so a tuple changed while they waited
no longer matches and the stale decision is skipped. _discard keeps the
equality compare-and-set rather than adding an expiry condition: its
early-release branch deletes a lock whose committed sentence has not
elapsed under the lowered live policy, which an expiry predicate would
wrongly block, and the equality predicate already pins every input of
the decision. reset stays unconditional by design.
Tests: a dialect-compile test asserts the DELETE's own WHERE carries the
expiry predicate for postgresql and sqlite (red on the previous HEAD), and
a PostgreSQL-only test (skipped without TEST_POSTGRES_URI) reproduces the
update-versus-sweep interleaving with two engines, waiting on
pg_stat_activity for the sweep to block before the extender commits.
Docs: sql.py concurrency contract and the migrations guide's 0034 bullet.
* fix(auth): keep an active login lock's clock and fail closed on explicit db
Address the "changes requested" review on #6501.
Lock clock (confirmed defect): record_failure re-stamped locked_at = now
for every failure on an already-locked row, so a sustained attacker's
sentence slid forward with each attempt. Both stores now apply one rule
in one atomic decision: new_count = fail_count + 1; below max_attempts the
IP counts with no lock (a failure after the operator raised the threshold
clears an existing lock); at or over it, a failure during an *active* lock
keeps locked_at and the committed lock_duration unchanged, while a
never-locked or served row starts anew with now / lockout_seconds (so the
failure that reaches a raised threshold starts a fresh lock rather than
keeping a stale locked_at). The SQL upsert expresses it as nested case()
on the row's own values; the memory store mirrors it. The sweeps in both
stores now leave the row being recorded to that decision instead of
deleting a served lock or stale counter first, which split the contract
(SQL would have produced a fresh (1, NULL, NULL) row). base.py states the
rule.
Explicit db downgrade: resolve_login_throttle_store raises SystemExit —
the failure type of the other startup validations in app.gateway.deps —
when throttle_storage='db' resolves to the shared table but no engine is
initialised, so langgraph_runtime refuses to start instead of silently
running every replica on per-process counters; the lazy bare-app path
applies the same rule and installs nothing. 'auto' keeps warning and
falling back. 'db' on a memory database still degrades with a warning
(there is no shared table to insist on).
Non-blocking items: _key() logs a warning when a client key is truncated
to 255 characters (never the key itself); the sql.py docstring covers
wall-clock locked_at across replicas (run NTP; skew shifts evaluation by
that much) and why STALE_COUNTER_SECONDS=24h / SWEEP_BATCH_SIZE=200 were
chosen; a store-level blocking-I/O anchor covers the SQL reset path; the
_validate_login_throttle_storage warning was verified to render
'throttle_storage=memory' for the StrEnum (the gate test now passes the
enum member and asserts that text). reset stays unconditional: a peer
failure that commits after a reset leaves fail_count=1, one genuine
failure after the cleared history, which a new test documents.
Docs: CHANGELOG entry, config.example.yaml, CONFIGURATION.md,
AUTH_DESIGN.md and the throttle_storage field description.
* fix(auth): re-evaluate a login-throttle check whose compare-and-set missed
SqlLoginThrottleStore.check applied its decision (commit the live
duration, or clear a served / released lock) with a compare-and-set
predicate on the snapshot it had read, but ignored the affected-row
count. When a peer changed the row between the read and the write — an
admitted wrong password incrementing fail_count on another replica, or a
successful login clearing the row — the UPDATE or DELETE matched zero
rows, yet check reported the decision as applied: it could return an
extended 60s sentence while the row kept its 1s duration, so the next
check cleared the lock and let password verification through; a live
duration decrease could be dropped the same way; and a row a peer had
just cleared was still reported as locked (#6501 review, P2).
Every compare-and-set write in check now checks result.rowcount. On a
miss the row is re-read and re-evaluated from the top, up to
CHECK_CAS_ATTEMPTS (3) times, applying a fresh compare-and-set decision
on the new snapshot and never clobbering a reset or replacement: a row
that is gone is reported as allowed, a fresh peer lock is reported as
locked, and an incremented count gets the extension or decrease
committed on it. Exhausting the bound fails closed: nothing is written
and the answer is the last snapshot's lock under the live policy, logged
at debug. The read stays in its own short transaction and each write in
a separate one.
The evaluation is now one shared pure function, evaluate_check in
base.py (CheckDecision), used by both stores so the contract cannot
diverge. The memory store keeps a single pass: its compare-and-set
cannot miss because there is no await between the read and the write on
the loop thread, which its docstring now states.
Tests (red first on df26c2acf) use a session proxy that runs the peer
mutation on a second engine right after the snapshot read ends its
transaction: the reviewer's raised-duration interleaving (59.5s reported
and committed, the increment survives, a later check still locked), the
mirrored decrease, early release on the fresh row, a reset landing
between read and write (0.0, no row recreated), and the bound-exhausted
fail-closed case.
* docs(changelog): resolve merge markers left in the Unreleased section
* test(migrations): pin 0035_login_throttle as the head in the 0034 watermark test
Upstream #6477 added test_migration_0034_run_event_seq_watermark.py asserting
0034 is the single head; this branch chains 0035_login_throttle after it, so
the test now pins the current head and walks the chain like its siblings.
* perf(auth): one session per login-throttle check, one transaction per failure
Two non-blocking performance suggestions from the #6501 review.
Single session per check. The router probed the row with store.get()
and then called store.check(), which re-read it: a recorded IP cost two
connection-pool checkouts and two SELECTs per request on the
unauthenticated login path, letting a lockout flood compete with
legitimate traffic for Postgres connections. LoginThrottleStore.check
now takes the policy as an async callable (LoginThrottlePolicy) and
resolves it lazily inside the store: the probe, the policy resolution
and the compare-and-set loop share one session, a clean IP returns 0.0
without ever calling the policy (the config.yaml read is still skipped
for the common case), and a recorded IP resolves it exactly once, also
across compare-and-set retries. The decision is made on a snapshot read
after the resolution: the policy read is a yield point, and a decision
that needs no write has no compare-and-set to catch a row a peer changed
meanwhile — so one session, but still a read after the yield. The
memory store takes the same signature and re-reads its record after the
await for the same reason. The router passes its asyncio.to_thread
policy read as the callable and drops the separate probe; get stays on
the protocol for tests and anchors. No short-TTL verdict cache: it would
mask a successful login's reset for the TTL window, and the single
session already halves the hot path.
Sweep in the upsert's transaction, plus an index. record_failure ran the
housekeeping DELETE in its own transaction and the upsert in a second
one, two commits (two WAL flushes on PostgreSQL) per failed login; the
sweep now rides in the upsert's transaction with one commit, the keep
exclusion already protecting the row being recorded. The new
ix_login_throttle_updated_at (ORM __table_args__ and the unreleased
0035 migration, created after the table and dropped before it, each
guarded separately so a table left by an interrupted attempt still gets
its index) serves the sweep's stale-counter predicate; the served-lock
predicate scans a table the sweep itself keeps small.
Tests (red first on a137fbc47): a clean IP never resolves the policy and
a recorded IP resolves it exactly once (both stores), the SQL store opens
exactly one session per check and commits once per record_failure
(counting wrappers), the migration test asserts the index after upgrade,
after completing a partial upgrade that lacked it, and gone after
downgrade while compare_metadata stays empty; the router-level
"mutated during policy resolution" tests now inject their mutation
through the policy callable.
* docs(changelog): note config_version 57 in the login throttle entry