mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-13 06:31:47 +00:00
339 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Wistom Zhang
|
2ff88fbb29
|
feat(memory): opt-in reserved categories for DeerMem hybrid-v1 eviction (#8194)
* feat(memory): support category capacity reservation in DeerMem hybrid-v1 eviction In long-running personal assistant interactions, core user profile facts (such as health allergies, coding style, or identity constraints) can suffer from capacity starvation as high-confidence conversational context accumulates. This change introduces opt-in category capacity reservation for DeerMem: - In DeerMemConfig and select_facts_for_capacity, support eviction_reserved_categories (defaulting to ['correction'] for historical compatibility) and eviction_decay_exempt_categories (defaulting to empty list). - Wire MemoryUpdater to pass the configured categories to select_facts_for_capacity. - Add deterministic offline unit tests in test_memory_personal_assistant_protection.py verifying root-cause reproduction, bidirectional lifecycle (clean explicit removal and contradiction replacement), and strict backward compatibility under defaults. * fix(memory): preserve correction priority in reserved capacity Co-authored-by: Wistom Zhang <77424093+Yakyou7@users.noreply.github.com> --------- Co-authored-by: Totoro <msy626836554@gmail.com> Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> |
||
|
Bas Cancrinus
|
13e6fc29e5
|
feat(opensandbox): add opt-in secure_access sandbox option (#8162)
* feat(opensandbox): add opt-in secure_access sandbox option Forward a new `sandbox.secure_access` option (default false) to SandboxSync.create so OpenSandbox can require a per-sandbox endpoint token at the ingress gateway and server proxy. The server returns the token as an endpoint header and SDK 0.1.15 already attaches endpoint headers to execd and file requests, so DeerFlow needs no token handling. Non-boolean values are rejected at provider startup so a quoted "false" cannot silently enable it. Documented in config.example.yaml, CONFIGURATION.md, the SandboxConfig docstring and the provider README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(opensandbox): validate server proxy boolean --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Totoro <msy626836554@gmail.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Wenchao An
|
9699f04cdc
|
feat(projects): add visible project memory with extraction controls (#8117)
* feat(projects): add visible project memory with extraction controls * fix(projects): avoid replaying committed memory extractions * chore(ci): retrigger project memory checks --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Wenchao An
|
3d826c6d95
|
feat(approvals): add Ask, Auto and Full tool permission modes (#8033)
* feat(hitl): add human-in-the-loop approval for high-risk tool calls * fix(hitl): address approval review findings * docs(hitl): keep agent guidance under size limit * fix(hitl): guard approval pause recovery * docs(hitl): shrink inherited guidance * fix(hitl): harden approval admission and recovery * fix(approvals): address review safety and reconciliation gaps * ci: retry approval checks after runner timeout --------- Co-authored-by: wangzeren <1004695029@qq.com> |
||
|
Wistom Zhang
|
218c64180e
|
feat(summarization): align built-in compaction prompt to preserve user requirements and approval conditions (#8114) (#8114)
In long-running agent workflows with repeated context compaction, empirical gateway ablations across 132 real sessions revealed that standard boundary compliance remains robust, while semantic drift can occur when model-generated summaries rephrase literal approval phrases and action conditions. This change converges PR #8114 into a clean built-in summary prompt enhancement: - Replaces regex extraction, tag elevation, and heuristic revocation with an extended built-in DeerFlow summary prompt (_DEERFLOW_SUMMARY_PROMPT). - Instructs the summarizer to faithfully record active user requirements, literal approval phrases, permitted actions, and later explicit revisions without paraphrasing. - Treats summary text strictly as historical data, fully preserving the security boundary and control/data plane separation of DurableContextMiddleware. - Unifies automatic and manual compaction factory paths to select the built-in template when unconfigured, while preserving explicit summary_prompt precedence. - Adds comprehensive sync/async unit tests covering built-in and custom templates. Co-authored-by: Totoro <Totoro-qaq@users.noreply.github.com> Co-authored-by: Totoro <msy626836554@gmail.com> |
||
|
Xinmin Zeng
|
2a97ec3e58
|
docs: turn README into an index and move reference detail to docs/guide (#8113)
* docs: turn README into an index and move reference detail to docs/guide The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md. * docs: send user docs to topic guides and budget the root README The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003). * chore: remove unreferenced screenshots and finished plan notes pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them. * docs: route always-loaded AGENTS.md files to scoped guides * docs(agents): cut scoped guides to scope, verify, traps, decisions Each AGENTS.md keeps four kinds of content: the scope of the directory, the verification commands, traps a local test run does not show, and decisions the code cannot state. Values, field lists, function inventories, test names, issue numbers and history are deleted, because the code, the tests and git state them. Search and fetch providers share one community guide. Four guides with no remaining content are removed. The size check now allows 6 KiB per file (hard limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB). * style: format frontend guide table and guidance test imports --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
SpecStream
|
fa8e015023
|
feat(community): add SpecStream web search and fetch provider (#7968)
* feat(community): add SpecStream web search and fetch provider * fix(specstream): keep a direct answer when sources are empty, add changelog entry Signed-off-by: Nataliia <nati@macbook-pro-nataliia.home> --------- Signed-off-by: Nataliia <nati@macbook-pro-nataliia.home> Co-authored-by: Specstreamai <333137486+Specstreamai@users.noreply.github.com> |
||
|
qkycir
|
f9767c65d5
|
fix(sandbox): run scoped env commands concurrently (#7841)
* fix(sandbox): run scoped env commands concurrently * test(sandbox): gate concurrent command admission deterministically * fix(sandbox): keep releasing scopes visible to client close * docs(sandbox): clarify scoped env ownership and capacity * fix(sandbox): retain cancellation ownership while scopes drain * fix(sandbox): bound Bash admission to provisioned session capacity |
||
|
Zeren Wang
|
0d04322d41
|
feat(hitl): add human-in-the-loop approval for high-risk tool calls (#7979)
* feat(hitl): add human-in-the-loop approval for high-risk tool calls * fix(hitl): address approval review findings * docs(hitl): keep agent guidance under size limit * fix(hitl): guard approval pause recovery * docs(hitl): shrink inherited guidance * fix(hitl): harden approval admission and recovery |
||
|
Eilen Shin
|
aaca3eeaeb
|
refactor(agent): ground runtime decisions in structured evidence (#7986)
* refactor(agent): ground runtime decisions in structured evidence Use producer status and structured errors for tool recovery, preserve memory context without lexical confirmation, and remove predetermined receipt and skill conclusions. Require explicit MCP discovery and authenticated schedule trial actions. Preserve authorization, ownership, isolation, and bounded execution; update documentation and regression coverage. * test(sandbox): expect read_file initialization error callback * fix(agent): address evidence review follow-ups * fix(agent): preserve retry budgets for OpenAI transport errors * fix(agent): expose tool execution status in model requests Project validated execution metadata and exact tool text into request-only JSON so status survives provider serialization. Preserve stored results and receipt hashes, cover SDK and graph boundaries, and migrate replay input hashes without changing recorded responses. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
ZhiLin Zhang
|
e5969fd8b9
|
feat(harness): introduce HarnessGuardMiddleware for pre-write syntax validation and anti-doom-loop (#7624)
* feat(harness): introduce HarnessGuardMiddleware for pre-write syntax validation and anti-doom-loop - Add HarnessGuardMiddleware to intercept tool writes and validate .py (AST) and .json syntax before disk write - Add anti-doom-loop state machine detecting repeated error signatures (>=2) and A -> B -> A -> B oscillating loops - Inject targeted remediation hints directly into tool results - Add DynamicSkillRouter for zero-LLM JIT skill intent activation (<1ms) - Fully unit-tested with 25 passing test cases * fix(harness): address review findings on write tools, scoping, and content handling * fix(harness): offload async validation off-thread, support Command results, and harden parsing * fix(harness): address reviewer 1 feedback on scoping, assembly count, and non-blocking verification - Revert .gitignore to exclude out-of-scope graphify-out entry - Short-circuit syntax validation when block_on_syntax_error is false - Trim AGENTS.md rule chain to fit hard guidance budget (AG002) - Fix input_sanitization mock and update subagent middleware count expectation to 23 * fix(harness): preserve chunked write flow, normalize CRLF line endings, and guard error classification - Preserve append-in-chunks flow: use codeop to distinguish incomplete Python fragments from syntax errors, and handle premature EOF on chunked JSON arrays/objects - Reuse native _match_line_endings for str_replace to prevent CRLF line ending bypass - Check existing tool failure status before classifying error text to avoid misclassifying successful tool outputs - Add 4 regression tests covering chunked writes, CRLF replacement, and successful error-text reads * fix(harness): scope pre-write syntax validation to str_replace and preserve write_file semantics * fix(harness): make str_replace syntax check opt-in, allow progressive repair of invalid files, and clean up RFC docs * chore: align subagent middleware count and satisfy AG002 guidance budget --------- Co-authored-by: Yakyou7 <Yakyou7@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
spud
|
21175ba46f
|
fix(runtime): persistently fence stale run event producers (#7619)
* fix(runtime): fence revoked run event producers * fix(runtime): reduce off-mode producer guard locking * docs: keep inherited guidance within size budget * chore(runtime): remove development phase labels * fix(persistence): normalize libpq SSL options for asyncpg connections * docs(runtime): clarify producer fencing behavior in off mode * test(persistence): align migration TLS probes with channel-binding rejection * fix(runtime): authorize fenced Gateway history seeding * fix(scheduler): preserve takeover identity in recovery publication * fix(scheduler): report unconfirmed recovery publication * fix(runtime): preserve history messages before the first human turn * fix(runtime): clarify run creation failure diagnostic --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
derrickLzw
|
51f0e2de1c
|
feat(extensions): scope configured middlewares to lead or subagent (#7661)
* feat(extensions): scope configured middlewares to lead or subagent * fix(helm): synchronize configured middleware config schema version * fix(extensions): require explicit configured middleware scope |
||
|
Totoro
|
8d11aeb97b
|
feat(subagents): configure thinking and reasoning effort for native workers (#7668)
* feat(subagents): add thinking_enabled and reasoning_effort config for sub-agents (#4875) (cherry picked from commit b1083728b23504cb0f1d1a463f8db70c5ec1c5a7) * fix: resolve CI failures for subagent thinking config PR - Fix UnboundLocalError in task_tool.py: initialize parent_thinking_enabled and parent_reasoning_effort before the if runtime block - Remove unused imports from test_subagent_thinking_config.py (SimpleNamespace, pytest, resolve_subagent_model_name) - Fix import sorting in subagents_config.py and test file - Add reasoning_effort: None to executor test assertion (cherry picked from commit 187851e307b871937e48bb67ae1b34b8b69bf92d) * fix: address review comments - remove parent inheritance, supports_thinking clamp, Literal type - Remove parent_thinking_enabled/parent_reasoning_effort from metadata extraction in task_tool.py (backward compatible: subagents only think when explicitly configured) - Add supports_thinking clamp in _create_agent to prevent cross-model mismatch errors - Change SubagentConfig.reasoning_effort to Literal type - Fix sandbox_state assignment that was accidentally removed - Fix syntax errors in subagents_config.py (missing closing parens) - Update all docstrings/comments from "inherit" to "no override" - Simplify test helper and remove parent inheritance tests (cherry picked from commit 74f9c659ef4a98bd1756109929ab0748fab82317) * feat(subagents): adapt reasoning settings to current native workers * fix(subagents): retain model context on reasoning rejection * fix(subagents): preserve model-factory reasoning diagnostics --------- Co-authored-by: Neon549 <neon549@users.noreply.github.com> Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> |
||
|
Agustin Torres
|
caafac0452
|
fix(wizard): replace retired Claude Sonnet 4 default with claude-sonnet-5-5 (#7585)
* fix(wizard): replace retired Claude Sonnet 4 default with claude-sonnet-5-5 The setup wizard's Anthropic default model is now `claude-sonnet-5-5` instead of the retired `claude-sonnet-4-20250514`. A new per-model `model_omitted_config` field makes the wizard leave out the extended-thinking (`budget_tokens`) and thinking-disabled templates for that model only, since Sonnet 5.5 rejects them; the other Claude models are unchanged. The Anthropic examples in `config.example.yaml` and `configuration.mdx` (which used `claude-3-5-sonnet-20241022`) now use `claude-sonnet-5-5` without those settings, and the change also adds a regression test and a CHANGELOG entry. Notice anthropic-retired-models-2026-10-27 about api.anthropic.com's API describes the change. It was read from a file and is not signed by the provider. * Update CHANGELOG with PR reference #7585 Added a reference to pull request #7585 in the CHANGELOG. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Tsai Yuan
|
89acf51caa
|
fix(tool-output): preserve prompt prefixes by default (#7455)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Totoro
|
ab8cd4e10f
|
fix(channels): keep IM binding codes on screen and report goal state in replies (#7591)
* fix(channels): keep IM binding codes on screen and report goal state in replies Connecting Slack, Discord, Feishu, DingTalk, WeCom, Buzz or QQ now opens a dialog that keeps the /connect command on screen with a copy button, its expiry time and the binding status, instead of a toast that disappears. The bots' /connect replies and the IM /goal replies now use the owner's web UI language (else channel_connections.notification_locale), and /goal reports the goal's state as the web goal bar does: in progress, why it stopped, or that it was met. * docs(changelog): note the IM binding dialog and localized IM replies (#7591) * test(channels): check IM reply tables against every notification locale The connect-reply check compared a fixed en-US/zh-CN pair, and the /goal reply table had none. Both now match NOTIFICATION_LOCALES with the same keys in each, like the existing _GOAL_REASON_TEXT check, so a new locale fails here instead of raising KeyError at reply time. --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> |
||
|
spud
|
aabfdeebc9
|
fix(streaming): make Redis stream END publication idempotent (#7572)
* fix(streaming): make Redis stream END publication idempotent * fix(streaming): align Redis test fake with Lua END checks Remove the unreachable owned missing-stream branch, mirror tail-only data checks in both fake script paths, and pin legacy END and owned missing-buffer behavior against real Redis. * fix(streaming): gate legacy END scans after writer rollout * fix(helm): synchronize embedded config version with schema * test(runtime): use real stream bridge config in close coverage Keep new configuration defaults in lifecycle fixtures and surface owner-task failures instead of masking them as event timeouts. Drain every helper task and cover context entry/close failures. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Totoro
|
c36bb699de
|
fix(channels): honor the configured recursion limit in IM runs (#7521)
* fix(channels): honor the configured recursion limit in IM runs Every IM channel run asked the Gateway for recursion_limit 100, because the channel manager seeded each run config with it. The Gateway honors a request value, so the top-level recursion_limit never reached IM runs, and a budget of 100 could stop a run after a handful of tool-calling turns (six in one measurement with the default lead-agent middleware; the count changes with the middleware chain). IM runs now send no recursion_limit of their own and get the Gateway's configured default, clamped by max_recursion_limit, like Gateway API runs. A session config value still takes precedence, and GitHub channel runs keep their floor of 250. The README and config.example.yaml IM examples no longer pin session.config.recursion_limit to 100. * docs(changelog): reference #7521 for the IM recursion-limit fix --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zeren Wang
|
74d2565f64
|
feat(projects): search shelf documents by keyword (#7286)
* feat(projects): search shelf documents by keyword Add search_project_documents: two-tier name/summary then bounded content scan, case-insensitive substring matching with original-text offsets. Never converts on the fly; reports stop_reason, content_scanned, skipped reasons, and capped unscanned/truncated ids. * fix(projects): derive unscanned count without paging after budget trip Review feedback: once the content budget trips, stop paging list_active to enumerate unscanned ids — cap ids at the trip page and derive the honest total from one count_active query (tier 1 is exhaustive, so active - tier1 hits - processed = unscanned). Also add a tool-level isolation test pinning that search never crosses projects or users. * fix(projects): scan the shelf over one stable snapshot, not offset pages Review feedback: offset paging re-read the live shelf per page, so a concurrent upload/trash between pages could duplicate a match or silently skip one while still claiming complete coverage. Tier 1 now materializes the whole shelf in a single SELECT (one read snapshot) and tier 2 iterates those same rows; the budget-trip unscanned count is derived exactly from the snapshot with no extra query. Mid-scan upload/trash regression tests pin both failure modes. |
||
|
Nan Gao
|
a9917e986b
|
docs(channels): align leader-lease failover and runtime-config notes with the merged code (#7420)
The two-Gateway harness checklist in CONFIGURATION.md said a crashed leader hands its providers over within lease_seconds + grace_seconds; a peer only tries to acquire on its fixed lease_seconds // 3 tick, so the worst case is one renewal interval more (50 s with the defaults), as the leader-lease section of the same file already says. The grace_seconds field description had the same shorter bound. The channels.leader_lease comment in config.example.yaml still said web UI credential changes reach the holder only when it next starts the channel; since #6988 the holder applies them within one renewal interval. The harness `check` footer still listed the IM channel gates as waiting for leader election, which landed in #6843; it now points at the manual channel item in CONFIGURATION.md, which also gains the runtime-config step. |
||
|
Lachlan Allen
|
7a32a5e177
|
feat(exa): support a configurable Exa-compatible base URL (#7306)
Exa web_search and web_fetch accept a per-tool base_url, falling back to EXA_BASE_URL and then the SDK default. Overrides are validated like the Serper base URL. web_fetch screens a configured backend with the shared delegated-backend guard. Co-authored-by: lachy3433 <lachlanallen.co@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Kiko0301
|
1a5f8edc2f
|
docs(config): explain skills and sandbox paths across platforms (#7304)
* docs(config): explain skills and sandbox paths across platforms * docs: clarify YAML paths and trim startup guidance --------- Co-authored-by: Kiko0301 <191240073+Kiko0301@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Paul Hunkin
|
27aa05f54e
|
feat(community): add Jsonify dataset rows tool (#7253)
* feat(community): add Jsonify dataset rows tool Adds deerflow.community.jsonify.tools:dataset_rows_tool, a read-only tool that returns rows from maintained Jsonify datasets. Jsonify pipelines run on a schedule and repair themselves when a source changes, so the agent reads a current table instead of scraping. - operator configures a named map of dataset -> serve URL; the model picks by name and never sees the URL - serve URLs must be https without userinfo and pass validate_public_http_url before any request; the URL is redacted from error text - paging with has_more/next_offset, version pinned after the first page, limit clamped to the configured max_rows, system columns stripped - config.example.yaml entry under tools; unit tests cover paging, clamping, unknown dataset, unconfigured tool, http/userinfo rejection, private address block and error redaction Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrmeK98ZKoxZNiUiz8kVeX * fix(community): harden Jsonify tool paging, logging and error handling - Stop paging on an empty page or a next_offset that does not advance - Log config validation failures by field location and error type only - Validate page shape; malformed or non-JSON pages return a clean error - Transport errors report the exception type, never the URL - Add truncated flag when rows were clipped or more rows remain - Tests for each case Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrmeK98ZKoxZNiUiz8kVeX --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Totoro
|
0baa99ca90
|
feat(models): add scoped model controls to Web chats (#7249)
* test: stabilize reporter evidence and guard missing chat nodes * feat(models): scope Web model choices to tasks, turns and conversations * test(e2e): mock model-control preview for ordinary chat submissions * fix(models): harden scoped choices and stale Agent submissions --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> |
||
|
李刚 / Bryan Li
|
1fd266e0c0
|
feat(decisions): add an OpenAI Decisions backend for the decision slots (#7127)
* feat(decisions): add an OpenAI Decisions backend for the decision slots Issue #7005 asked for a second decision provider behind the interfaces the TypeSafe (Jev) integration already uses, not a new decision system. `deerflow/openai_decisions` speaks `POST /v1/decisions`: a `noul` question becomes a `predicate` whose true/false rubric folds into `instructions` (the endpoint has no criteria field, and dropping it would move the decision boundary while the adapter kept reporting the same criteria in its policy identity), answers arrive as an array rather than a mapping, and a refusal is an answer `type` rather than an HTTP failure -- so it lands in the existing per-question error channel and each consumer keeps its own failure direction. Transport is reused rather than copied. `provider_label` and the connection's `endpoint_path` are the only two seams a dialect needs, so retry and backoff, the deadline budget, the bounded never-decoded body read and the error taxonomy keep one owner; `resolve_decisions_connection` pairs the two so a Decisions connection cannot be built pointing at System One. The three adapters override their connection resolution, their client, and the value the endpoint judges (`input` text instead of a `state` object), and inherit thresholds, rubrics, caches and audit policy unchanged. Wiring stays a `use:` class path, so no config schema is added. * fix(decisions): isolate endpoint identities and correct diagnostics --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
RongJie G
|
bf2cb0e1df
|
fix(runtime): recover terminal Redis streams after producer loss (#7093)
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Ryker_Feng
|
d1d0958bea
|
feat(skills): add guarded evolution host APIs (#5645)
* feat(skills): add guarded evolution host APIs * docs(agents): avoid growing saturated guidance chains * fix(skills): address mutation host review findings * fix(skills): order name fence before projection lock * fix(skills): preserve composite toggle lock order * fix(skills): address catalog and pagination review feedback * docs: keep subagent guidance within merged instruction budget --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Totoro
|
16d36ccd21
|
feat(sandbox): add bounded AIO prewarming and lifecycle benchmarks (#6998)
* fix(benchmark): validate sandbox lifecycle and concurrent measurements * feat(sandbox): prewarm likely AIO tool turns (cherry picked from commit 3a1017ed032ed9331987bc3dc61f23dcdac4e905) * fix(sandbox): bind bounded prewarming to execution leases * docs: trim frontend guidance below its existing size limit * fix(benchmark): separate failed lifecycle stage timings * test(sandbox): measure prewarm overlap and unused resources through agent graphs * fix(sandbox): finish lease and sampler cleanup after cancellation or errors * docs(benchmark): distinguish initialization wait from model overlap * docs(helm): align the configuration example with version 61 * fix(sandbox): preserve synchronous agent graphs when prewarming is enabled * test(cargo): isolate native fixtures from inherited make jobserver flags * fix(sandbox): require managed cleanup before prewarming * fix(sandbox): log denied prewarming as an expected skip * fix(benchmark): filter provider summary input records --------- Co-authored-by: nothingyuancando <1579035512@qq.com> |
||
|
Nan Gao
|
e8c7df9042
|
feat(channels): arbitrate IM transports with a per-provider leader lease (#6843)
* feat(channels): arbitrate IM transports with a per-provider leader lease Every Gateway instance connected every enabled IM platform with the same bot credentials, so channels could not run on more than one instance (Telegram 409 storms, Discord answering twice, scattered per-process state). channels.leader_lease was reserved by #6746; this implements it. - channel_leases table (migration 0041): one row per provider. Acquire is one dialect-native upsert decided against the database clock and bumps a fencing epoch; renew extends only a live lease at the holder's epoch; release sets the expiry to 0 so a peer takes over on its next tick. - app/channels/leadership.py: the holder's local monotonic deadline (send time of the last confirmed renewal + lease_seconds); holds() turns false one renewal interval early, so the transport stops at least grace_seconds before a peer may connect. DB calls are bounded. - ChannelService: start fence in _start_channel (no DB call) behind every start path; fixed-rate supervisor renews/acquires and runs transport starts/stops as per-provider tasks; takeover reloads the provider's saved config; failed starts back off; shutdown releases only leases whose transport stopped. get_status reports leader/leader_here and the holder's transport state on every instance. - Notification worker leaves rows of providers another instance holds. - Gate: the enabled=false refusal now recommends auto/true; an explicit true with database.backend memory is refused when a channel is enabled. - Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated. * docs(changelog): link the channel leader lease entry to #6843 * fix(channels): see peer-held leases for providers absent from local config Review on #6843: a provider connected from the web UI on a peer never enters this instance's channel config, so the notification worker here still claimed its rows (and parked them for 900s) and GET /api/channels/ reported no leader for it. Every supervision round, and startup, now also reads the whole lease table once (bounded), and notification exclusions and status use those observations: a provider whose live lease a peer holds is left to it even without local config. Running that read concurrently with renewals exposed that, through SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on another connection can hand back one of its own bind values instead of the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do not show it). A lease must never take such a row for a confirmed renewal, so acquire/renew/release now decide by affected-row count and acquire reads its epoch back in the same transaction. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zhengxu (Joshua) Jin
|
70e8f8974f
|
fix(channels): parse Slack and DingTalk allowed_users fail-closed (#6742)
* fix(channels): parse Slack and DingTalk allowed_users fail-closed Slack and DingTalk still built their allowlists with their own _normalize_allowed_users: every entry went through str() without stripping, only list/tuple/set counted as collections, and the gate was `if self._allowed_users and ...`. A configured list whose entries are all blank, such as `allowed_users: [$SLACK_USER_ID]` with the variable set to an empty string, became an empty set and let every sender through. " U123 " never matched U123, and a frozenset was stringified into one ID. Parse both with the shared parse_allowed_users helper that Telegram, WeChat and QQ use, and gate on `is not None`: unset, null, empty collections and a blank string allow everyone; a scalar is one entry; strings are stripped and integers keep matching their string form, as before; booleans, floats, None and mappings are rejected (a mapping as a whole); a configured value with no valid ID denies every sender and logs an error. A scalar containing commas or whitespace warns that it stays one literal ID. * refactor(channels): share the Slack and DingTalk allowlist coercer and separator warning Slack and DingTalk had byte-identical ID coercers and their own copies of the scalar-separator warning. Move the coercer into allowed_users.py as parse_string_or_int_user_id and opt both channels into warn_on_scalar_separators, as QQ and WeChat do. Log messages are unchanged. Pin that a separator-only string such as "," stays one literal ID (denying everyone else) and still warns. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
jay
|
eecfebd419
|
fix(buzz): treat a scalar allowed_users entry as one member pubkey (#6811)
* fix(buzz): treat a scalar allowed_users entry as one member pubkey
BuzzChannel built its allowlist by iterating config.get("allowed_users") directly. A scalar string -- what an allowed_users: $BUZZ_OWNER_PUBKEY reference resolves to -- was iterated character by character, feeding each character to parse_pubkey, whose first non-hex character raised and aborted channel construction; a numeric scalar raised TypeError the same way. The sibling QQ (#6730) and Discord (#6776) connectors already treat a scalar as a single entry.
Parse the value with the shared parse_allowed_users helper via a parse_pubkey coercer: a scalar is one entry, list entries keep their per-entry validation, unusable entries (bad hex, numeric refs, mappings) are dropped with the shared warning instead of crashing, and the deny-by-default contract is unchanged -- an empty or blank value still yields an empty allowlist with start()'s existing deny-all warning.
config.example.yaml states the accepted shape next to the key, matching the mention_free_channels note added by #6776.
* docs(config): split buzz allowed_users note into aligned comment lines
The two-line note on the buzz example entry was emitted with a literal
`/n` where a newline was intended, so the second note never became its
own comment line and the `#`-column alignment shared with the
DENY-BY-DEFAULT lines below was broken. Split it to match the
mention_free_channels sibling shape (review comment on #6811).
|
||
|
OUAO-FRANK
|
12397d2142
|
fix(sandbox): forward bash command timeout to remote providers (#6782) | ||
|
Grapette.L
|
9b2151a3a8
|
fix(buzz): a single mention_free_channels ID was split into characters, a numeric one stopped the connector (#6776)
* fix(buzz): accept a single mention-free channel id `_mention_free` iterated the configured value directly, so one unquoted UUID became a set of its characters and a numeric or boolean scalar raised TypeError while the channel was being constructed. Both shapes are documented as valid for the sibling Discord connector. Parse the key through a dedicated helper that follows the Discord convention: a scalar is a single entry, list entries are kept verbatim, and an unusable entry never matches so `require_mention` stays in force. * fix(channels): share scalar-or-list channel ID parsing * docs(channels): keep guidance within CI size budget --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
d18d2937e7
|
feat(channels): add channels.leader_lease and gate an explicitly disabled lease under multi-process signals (#6746)
* feat(channels): add channels.leader_lease and gate it under multi-process signals IM channels other than GitHub connect outbound to their platform. With more than one Gateway instance each one opens that connection with the same bot credentials: Telegram rejects the second poller with 409, Discord answers every message twice and per-process ChannelManager state diverges. The multi-process startup gate said nothing about channels; the only guard was a comment in the Helm values. Add the `channels.leader_lease` section (`enabled: auto|true|false`, `lease_seconds`, `grace_seconds`; startup-only, registered in the reload boundary) that the per-provider leader lease will read, validated by a Pydantic model from the non-schema `channels` mapping. The gate now refuses `enabled: false` under a multi-process signal while a non-GitHub provider is enabled under `channels.*` or connectable through `channel_connections`, naming the providers and the rollback; a malformed block is refused on every topology. `auto` follows the multi-process signal, so a single instance is unchanged. ChannelService strips the block before its per-provider loops and exposes it as `leader_lease`; `ensure_ready_channels` only visits registered providers. The two-Gateway harness overlay leaves the block alone. config_version 60 in the example, the Helm chart and its README; docs replace the comment-only "keep 1 replica" guidance with the gate. * docs(changelog): link #6746 * fix(channels): stop recommending the unimplemented leader lease as a remedy Review on #6746: the refusal told operators to switch channels.leader_lease from false to auto/true "so one instance owns each provider's connection", but the transport arbitration ships separately, so that change only bypassed the clause and every worker still started the same bot. The refusal now offers only the remedies that work today (a single instance, or disabling the channels) and states that auto/true are reserved for the lease that ships separately. A regression helper in the gate tests asserts the remedies start with the rollback and that auto/true are never offered. The config example, Helm values/README, CONFIGURATION.md, README, the deployment guide (en/zh), the model docstrings and both changelogs describe auto/true as reserved settings rather than an ownership guarantee. * fix(channels): warn on an enabled unknown channel key; reword the changelog entry Review on #6746: the registry guard in ensure_ready_channels skipped a misspelled provider key (channels.telegam) silently, where it previously reached _start_channel and logged 'Unknown channel type'. An enabled dict block that is neither a registered provider nor leader_lease now logs a warning that lists the registry. The changelog entry claimed the chart and docs drop the "keep 1 replica" guidance; they keep it (auto/true do not make several replicas safe yet) and only stop relying on the comment alone. * fix(dev): disable UI channel providers in multi-instance overlay --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zhengxu (Joshua) Jin
|
47b6755ce9
|
fix(qq): parse allowed_users with the shared fail-closed allowlist (#6730)
* fix(qq): parse allowed_users with the shared fail-closed allowlist
QQChannel built its allowlist with set(config.get("allowed_users") or []).
A single OpenID, which is what an `allowed_users: $QQ_ALLOWED_USERS`
reference resolves to, became a set of its characters, so the configured
user was silently dropped. A mapping authorized every key, including one
mapped to false, and a numeric scalar raised TypeError while the channel
was being constructed.
Parse it with the shared parse_allowed_users helper that Telegram and
WeChat already use: unset, null, empty collections and a blank string
allow everyone; a scalar OpenID is one entry; entries that are not
non-blank strings (such as numeric QQ account numbers) are dropped with a
warning, and a mapping is rejected as a whole; a configured value with no
valid OpenID denies every sender and logs an error. A scalar containing commas or whitespace warns
that it stays one literal ID. Binding codes still precede the allowlist.
* test(qq): gate group messages by member OpenID in allowlist tests
Every allowlist test sent a C2C frame, and the shared event() helper puts
the sender in both user_openid and member_openid, so a group-path
regression in sender resolution would pass unnoticed. admits() now takes
a kind; group frames carry only member_openid and a group_openid that
differs from the sender. The listed-OpenID test runs for both kinds, and a
new test shows that listing the group's OpenID does not admit its members.
Document that a blank or whitespace-only allowed_users string allows every
sender, like an unset value or an empty list.
* refactor(channels): share scalar allowlist separator warnings
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Storm
|
3f7a076972
|
feat(channels): Feishu interactive clarification cards (#6670)
* feat(channels): render Feishu option clarifications as button cards When the agent's ask_clarification carries options, the Feishu channel now renders the question as an interactive card with one button per option instead of plain text. This is the outbound half: card construction plus a process-local pending registry that a later card-action callback handler will validate clicks against. - New feishu_card_interaction module owns request extraction, button card rendering, and the pending registry (30-minute TTL, matching the typed-reply clarification window) - Manager attaches the clarification's structured human_input payload to the final outbound's metadata so channels can render richer affordances without touching the Gateway protocol - Feishu channel renders the button card only when card_interaction_enabled is set (default off); free-text, form, and over-four-option clarifications keep the plain-text path - Button cards bypass the markdown wrapper via a raw_card_content flag on the card send/patch helpers; plain-text call signatures unchanged * feat(channels): close the Feishu button-card click loop The inbound half of interactive clarification cards: a click on an option button validates, resolves the clarification, and marks the card in place. - Register the card-action callback only when card_interaction_enabled is set; clicks arrive as WS EVENT frames through the SDK's callback processor map, and the synchronous return value becomes the toast ack - consume_if_originator atomically validates operator identity against the pending entry: non-originators get a refusal toast and the entry survives; unknown or expired requests are toast-only and dropped - A validated click consumes once, updates the card to a terminal resolved state, and injects a hidden inbound carrying the human_input_response so the run resumes on the originating thread with Web-card parity - Manager translates the inbound's human_input_response metadata into the human message's additional_kwargs with hide_from_ui set * docs(channels): document Feishu interactive clarification cards - README / README_zh: optional card_interaction_enabled setup, calling out that card.action.trigger lives under the console's callback configuration tab (separate from event subscriptions) and requires publishing a new app version - config.example.yaml: commented flag with the same console caveat - backend/AGENTS.md: module ownership note for feishu_card_interaction * fix(channels): close Feishu card-click routing and ack gaps from review Address review on #6670: - Carry the routing identity (topic_id, connection_id, owner_user_id) on the pending entry and the synthetic inbound, so clicks resolve to the clarified thread in topic-scoped chats and pass bound-identity admission instead of opening a new thread or being rejected - Validate option_value against the registered options at consume time: an unknown value is treated like a stale card and the entry is kept - Return a warning toast when a click cannot be dispatched, and flip the card to its resolved state only after the inbound is durably in the bus — never display a choice that was not recorded - Drop the dead mark_sent/consume registry APIs and pass the payload's own option ids through to the buttons - Add an end-to-end test through the manager's thread resolution pinning that a card answer reuses the clarified thread * fix(channels): carry workspace_id on the Feishu card-click inbound The connections-enabled admission path re-reads the binding with the message's workspace_id, and Feishu connections are bound with workspace_id = chat_id. The synthetic inbound set connection_id and owner_user_id but never workspace_id, so the re-read missed and every click was rejected as unbound after the entry was consumed. - Set inbound.workspace_id from the entry's chat_id (Feishu's chat-as-workspace binding convention) - Extend the card-click e2e to require_bound_identity=True, driven through the channel's real _resolve_card_click production * fix(channels): stabilize Feishu clarification delivery and routing Option cards could fail through the real reply SDK, resume another conversation, or leave a stale text hint after the answer was accepted. Unify raw-card delivery and persist trusted thread aliases before showing buttons. Keep protocol request IDs separate from bounded, one-time callback tokens; validate originators and option pairs, and retire only the accepted card's text/button state. Treat valid hidden human answers as turn boundaries so old replies and artifacts do not leak into the next answer. Cover consecutive clarifications and /new routing through real SDK request models, stores, and the run journal. Validation: 147 focused tests and 29,421 offline tests passed (405 skipped); strict blocking-I/O passed 245 tests (1 skipped). Ruff and guidance checks passed. * fix(channels): preserve clarification context in Feishu cards Button cards replaced the full clarification text but omitted its background, leaving users without the context needed to choose an option. Carry optional context from request metadata and render its original Markdown before the question. Keep question-only cards for absent, blank, or invalid context. Cover rendering and all real SDK delivery paths, and update both user guides. Validation: 156 focused tests, 245 strict blocking-I/O tests, Ruff, and guidance checks passed. The offline run recorded 29,381 passed, 49 failed, and 405 skipped; 48 import failures reproduced before the fix. All 49 cases passed on both versions in isolated environments, and the Feishu-to-export order check passed 13 tests. |
||
|
Jie Ji
|
bdc2d12367
|
fix(memory): prevent cross-conversation debounce starvation (#6672)
* fix(memory): prevent cross-conversation debounce starvation * fix(config): bump schema version for memory max wait |
||
|
Value Add
|
4beb6c4ea4
|
feat(community): add You.com web search provider (#6515)
* feat(community): add You.com web search provider Adds You.com as an opt-in community `web_search` engine, mirroring the existing community engines (tavily, brave, exa, firecrawl, groundroute). You.com runs its own web index and returns extracted page snippets per hit rather than a single meta description, so a research loop gets usable context without a follow-up fetch. The provider picks its endpoint by whether a key is configured, so it works out of the box and scales up when the operator adds one: - no key -> /v1/agents/search, keyless free tier - YDC_API_KEY -> /v1/search with X-API-Key, full Search API Endpoint and headers are always chosen together; the keyless endpoint rejects an auth header, so a stale key can never be sent there. Both return the same body shape, so one normalizer covers both. httpx only, no new dependency and no pyproject extra, so uv.lock is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint * fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY The doctor detail returned before any key check, so operators who had set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'. Reworded to match the sibling free-provider entries and stay accurate in both states. * fix(search): sweep youcom max_results onto the shared owner The single-owner gate that landed with #6563 flags this branch's private _coerce_max_results as a newly hand-rolled copy, so backend-unit-tests now fails after merging main. Follow the coordination note from review: delegate to deerflow.community.search_max_results.coerce_max_results, which owns the rejection profile (booleans, non-integral floats, OverflowError), and keep only the 1-100 mirror of You.com's server-side count clamp on top — clamping policy stays per-provider. The .inf regression test now goes through the public tool path and also pins the bool and non-integral-float shapes of the shared bar. * docs(tools): use the bare you.com platform link in config and docs Match every other provider entry: no other provider in config.example.yaml or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params from the three you.com/platform links. Integration attribution stays in the identifying User-Agent sent to You.com hosts. * test(youcom): cover shared max_results ownership and bounds Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters. --------- Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
yang rui
|
a333ac3189
|
feat(search): add Webz news search provider and setup integration (#5880)
* feat(search): add Webz news search provider and setup integration * fix(webz): honor caller result limits and clarify page counts * fix(frontend): render Webz search source links * fix(frontend): skip malformed search source entries |
||
|
yangfucheng123
|
3b8141aa43
|
fix(community): Fail closed on delegated fetch backends to close SSRF gap (#6531)
* Fail closed on delegated fetch backends to close SSRF gap * fix(community): resolve delegated-fetch credentials from one config snapshot Address review feedback on the fail-closed SSRF guard: - firecrawl/fastcrw client construction no longer re-reads get_app_config(), so the API key and endpoint come from the same revision the endpoint screen validated (prevents cross-revision credential misrouting). - fix the browserless test double to accept the new two-argument _get_browserless_client signature. - document the check-only (DNS rebinding) caveat on validate_delegated_backend_url. - add a CHANGELOG upgrade note for the default self-hosted backend break. * fix(doctor): consolidate delegated fetch backend checks and deployment guidance --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
creed
|
be62fff070
|
feat(serper): add opt-in bounded transient retries (#6456)
* feat(serper): add opt-in bounded transient retries Signed-off-by: 97three <2212371308@qq.com> * fix(serper): validate HTTP retry dates precisely Signed-off-by: 97three <2212371308@qq.com> * fix(serper): address retry review feedback --------- Signed-off-by: 97three <2212371308@qq.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zeren Wang
|
553f85e788
|
feat(projects): best-effort LLM summaries for shelf documents (#6581)
Opt-in per-document summary column rendered into the injected <documents> index, list_project_documents JSON, and the documents API. Pipeline (projects/summaries.py, behind the narrow enqueue_summary interface): - The two shelf write routes enqueue a job on created=True; dedup hits never regenerate. Off by default (projects.summaries_enabled). - Bounded worker pool (summary_concurrency + bounded summary_queue_size; a full queue drops the job) fed by FIFO; shutdown drains queued jobs, delivers one sentinel per worker via blocking put (queue smaller than the worker pool cannot strand idle workers), and bounds only genuinely in-flight jobs with the grace period. - Jobs carry (document_id, project_id, user_id) explicitly; the job's owner is bound into the user ContextVar around the model call so tracing/usage attribution never falls back to 'default'. - Text source reuses read_text_serving_path (convertible-first rule, content validation); conversion has no deadline (await_drained), the model call has a caller-side wait limit — the worker slot is retained until the provider call actually finishes and late results are discarded. - Output contract: strip leading <think> reasoning -> unwrap fence/ quotes -> collapse to one line -> empty is NULL -> UTF-8 byte cap on a character boundary; length-capped responses (provider-normalized detectors) are failures. - set_summary pins updated_at (literal self-assignment) so a landing summary never reorders the shelf; owner-predicated UPDATE is a no-op on rows purged mid-generation. - Metrics: eligible/success/failed/skipped/queue_dropped/ uncapped_accepted. Schema/config: - project_documents.summary (migration 0036, nullable, no backfill; forward-compat floor unchanged). - Six projects.summary_* keys, all registered in STARTUP_ONLY_FIELDS (captured once at lifespan startup); config_version 57 -> 58. - run_oneshot_llm gains an optional model_overrides passthrough (passed only when set, existing callers unchanged) and a detailed variant exposing the raw response message for termination-signal detection. Docs: API.md response field, ARCHITECTURE.md pipeline paragraph, migrations AGENTS.md chain head, root AGENTS.md records the scoped all-core exception with the lift-out path. |
||
|
Totoro
|
0fe6e63a68
|
feat(goal): show why a chat goal stopped and when it was met, and redact the goal evaluator and continuation input (#6556)
* feat(goal): record the met goal and keep goal state on the history head - Add a goal_outcome LastValue channel (GoalOutcomeState). Only the guarded clear of a satisfied ordinary goal writes it, in the same checkpoint as the clear; the reply anchor is the latest visible AI message with text, the reply the visible signature keys on. A goal stored without created_at still clears and gets a record. - write_thread_goal removes the record on every other goal write and lists it in channel_versions, new_versions and metadata writes when it changes. Threads that never had a record keep their checkpoint shape. The sync-only SqliteSaver and CachedHistorySaver, which take the #6448 sync fallback, store and drop it the same way. - goal_outcome is server-owned: POST /state and non-internal run input that carry it get 400. A POST /state goal write, or a goal in external run input, clears it. - A branch head written onto its replay base resets the channels the head lacks, so a goal or record that write_thread_goal removed does not come back. Branching from the met turn keeps the record without the met goal. - The POST /history head now carries an active goal and an achieved goal_outcome, also before any message exists, so the post-run history refetch keeps them. Older entries stay title/thread_data. - contracts/thread_goal_contract.json pins the chat stand-down and check-failure codes, the record shape and status, the head keys and the host-written reasons. * feat(frontend): show why a goal stopped and when it was met - The goal bar derives its state from the active goal, the goal_outcome record and the messages: set, in progress, continuing, stopped (limit, auto-continue off, no progress, token budget, missing evidence, failed run, unknown code), waiting for you, waiting, couldn't check the goal, paused and goal met. An 8/8 stand-down now reads "Stopped" with the next step instead of "Continuing 8/8", and a count-0 stand-down no longer looks like a fresh goal. - "Goal met" shows while its anchor reply is the latest visible message (a visible human message, or a visible AI message with text, moves the chat on), so a reload agrees with the live view. An active goal wins over the record, and this tab's /goal set or clear hides it. - useActiveGoal drops the optimistic goal when the record of that same goal arrives, which fixes the bar that stayed after a goal met on the first run (the clear frame has no goal key), also for a goal stored without created_at. Server goal values pass through isGoalState; a /goal status read uses the backend's own active check, so the bar, the toast and the edit lock agree with the server. - Row 2 gives the reason and what to do next; Details holds the full goal, the checker's own note (never host-written text, a check failure or a paused goal's older check), the assumption note and an unknown code, and starts collapsed in each new state. The bar is a labelled section with a live region that announces state changes only (another goal's same state counts) and re-reads its text after a language switch. - The edit pencil on the latest turn is shown disabled with "Editing is off while a goal is set" instead of hidden; the message toolbar shows on focus-within, so keyboard users can reach it. - A /goal set or clear rejected with 409 keeps the draft and says a run is still going; other failures keep the server's message and the draft. A /goal request replaced by a newer one leaves the composer alone, so a double submit no longer empties it before the second one fails. - Copy lives in inputBox.goalBar (en-US, zh-CN); a contract test reads contracts/thread_goal_contract.json, and the raw-identifier guard now catches stand-down codes and goal field names. * test(frontend): cover the goal bar on both chat pages - tests/e2e/goal-status.spec.ts runs every case on the chat page and the agent chat page, since each wires the goal bar and the edit lock itself: a goal stopped at 8/8, one waiting for details and one waiting on an open question card show their reason and next step, also after a reload; the locked edit pencil gets keyboard focus, shows its toolbar and tooltip, does not open the editor, and hides while a file uploads; Goal met after one automatic follow-up (In progress, Continuing 1/8, then "Auto-continued once") survives a reload and hides after the next visible message; a goal met on the first run of a new chat replaces the set bar and frees the pencil, and a new /goal replaces Goal met while its own run goes on; a /goal set or clear refused with 409 keeps the draft for the retry and shows its own busy toast. - The mock POST /history head carries only the contract's history_head_keys, with goal only while active and goal_outcome only when achieved; thread state and run values omit an unset goal instead of sending goal: null. PUT and DELETE /goal drop the met record, setGoalWritesBusy returns the Gateway's 409, and a repeat create of a requested thread id returns the thread as is, like the Gateway. - The raw-identifier guard also catches checker blockers and the met record's field names. - frontend/src/AGENTS.md: the GoalStatus sentence now names goal, goal_outcome and useActiveGoal, reworded in place. * test(frontend): keep the draft through a failed upload and a busy /goal #6412 fires onSent only once a send is dispatched, so a failed upload keeps the text, files and stored draft. A /goal the Gateway refuses with 409 never reaches sendMessage, and the composer rejects it so PromptInput keeps the text. Drive both through the real sendMessage in one composer: a failed upload keeps "summarize it" stored, a busy /goal keeps its text and stored draft with no run, and the retry saves the goal and starts one run with the objective, not the "/goal" text. After that run the composer is empty and no stored draft comes back. * fix(frontend): drop the stop condition's closing period before the pause clause A stored stop condition often ends with a period, so the schedule card read "…remain)., pauses itself" (and "…。，满足后自动暂停" in Chinese). Trim the condition's closing sentence punctuation before the clause that follows it. The event line, where the condition ends the sentence, is unchanged. * fix(frontend): point "evidence missing" at the agent, not a user reply "Evidence missing" asked the user to reply with what's left, which is the push a goal is meant to save. It now asks the agent to show the result or explain what's missing, and offers /goal clear only for a goal that no longer applies: clearing drops the met record, so work that is right would never show "Goal met". zh-CN follows. Copy only: no key, state or mapping changes, and the scheduled page's reason labels are untouched. tests/unit/components/workspace/goal-status-next-steps.test.ts checks the row in en-US and zh-CN, and that no stop row offers /goal clear without that condition. goal-status.dom.test.tsx and goal-stream.dom.test.tsx now expect the new copy. * fix(frontend): say "Couldn't check the goal" once in the goal bar After a goal check that could not run, the bar's second line borrowed the scheduled page's whole reason, so it repeated the chip: "Couldn't check the goal · Couldn't check the goal; the run itself may be fine. It's checked again after your next message." Chinese had the same repeat. The line now gives only why the check failed ("The run itself may be fine", "No final reply was saved" or "The chat changed during the check"), then when it is checked again. The screen-reader announcement follows. The chip and the scheduled page's reason labels are unchanged. tests/unit/components/workspace/goal-status-unchecked.test.ts checks every check-failure code in en-US and zh-CN: the label appears once, and the line still says why the check failed. * feat(goal): redact the evaluator input when pii_redaction is enabled The goal evaluator calls its model directly, outside PiiRedactionMiddleware, so it sent raw user lines, tool arguments, tool results and Human Input Card answers even with pii_redaction enabled. With pii_redaction enabled: - the messages in the evidence window are redacted whole before the evidence caps can cut an identifier in two, as TitleMiddleware does. This reuses redact_queued_messages, which already covers content, call arguments and card answers. Earlier messages are left alone: the evaluator never sees them, and the pass is synchronous work on the event loop (0.84 s for an 800-message, 4 MB thread, 0.06 s for its window); - the assembled input, the goal objective included, is redacted again. A redaction error fails the check with evaluator_failed; raw text is never sent instead. When redaction is off, the input is byte-identical. Thread state keeps the raw text. README.md, config.example.yaml and the middleware docstring list the evaluator with the other redacted inputs. tests/test_goal_pii.py puts an address where the evidence caps cut a tool argument, a tool result and a card answer, and checks the objective, the unchanged input when redaction is off, that only the evidence window reaches the redactor and gives the same evidence as redacting the whole thread, that a redaction error stands down with evaluator_failed before the model is called, and the worker path. * fix(goal): redact the hidden continuation when pii_redaction is enabled The hidden /goal continuation repeats the goal objective and the evaluator's reason and evidence summary. PiiRedactionMiddleware skips it, since it is a framework message and not a user message, so with pii_redaction enabled each continuation turn sent them raw to the agent's model, and the message stayed in the thread for later model calls. The durable-context block and the evaluator input already redact the objective. With pii_redaction enabled, make_goal_continuation_message now redacts the objective, the reason and the evidence summary with redact_text before the message is built. The thread keeps the redacted message, which the UI hides; the goal state keeps the raw objective. The worker builds the message before it counts the continuation. A redaction error fails the check the way an evaluator redaction error does: run_failed with stand_down_reason evaluator_failed, the goal stays active, nothing is sent and the continuation budget is not spent. Only the exception type is stored. With redaction off, the message is byte-identical. README.md, config.example.yaml and the middleware docstring list the continuation with the other redacted inputs. tests/test_goal_continuation_pii.py checks the redacted objective, reason and evidence summary against the tokens redact_text gives, the unchanged message with redaction off or no config, the worker path under the run config, and a redaction error. * docs(changelog): link the goal outcome entries to #6556 Fill the four ([#PR]) placeholders in CHANGELOG.md and CHANGELOG_zh.md with this PR's number and add the matching link references. * fix(frontend): keep an active goal stored without timestamps visible POST /threads/{id}/state and run input can store an active goal without created_at or updated_at. The backend still evaluates it and rejects an edit with 409 (is_active_goal checks only status "active"), but isGoalState required both timestamps, so history and live values dropped it: both chat pages hid the goal bar and unlocked the last-turn pencil, and only a /goal status read, which used a looser predicate, brought them back. isGoalState now accepts any record with a string objective and status "active", and normalizeGoalStatusRead uses it, so thread values, live updates and status reads share one rule. Malformed or inactive values are still dropped. Tests cover a sparse active goal from the history head and from a live update before the post-run refetch (bar shown, hasGoal true), and a malformed goal that stays hidden. * test(goal): check the outcome write metadata the same way on every saver PostgresSaver stores checkpoint metadata through get_serializable_checkpoint_metadata, which drops "writes", so the [postgres] case of test_goal_outcome_round_trips_through_the_sync_fallback failed with KeyError: 'writes' when CI ran it against a live database. The test now records the metadata write_thread_goal passes to put and asserts the clear's and the next goal's "writes" there, for every saver. The stored channel values are checked as before, and on sqlite and cached-sqlite the stored "writes" must still equal what was passed. * docs(agents): keep goal_outcome out of the agents guidance chain The middlewares AGENTS.md chain is already over the 98,304-byte AG002 hard limit on main, and the PR-mode check fails any chain a PR touches, so adding goal_outcome to the ThreadState field list in deerflow/agents/AGENTS.md failed the agent-guidance job. goal_outcome stays documented in runtime/AGENTS.md and backend/app/gateway/AGENTS.md, which are outside that chain. * refactor(goal): read the evaluator's evidence and redaction from one window format_visible_conversation recomputed the window start that the redaction pass takes from _evidence_window. Both now use _evidence_window, so a later change to the cap or to what counts as visible moves both. The window test now fills the window with tool-only replies and tool results. Before, it caught a cap change in one copy but not a change to what counts as visible. --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
9c159ef880
|
fix(auth): key the compose login lockout by client, not by nginx (#6519)
* fix(auth): key the compose login lockout by client, not by nginx Failed logins are counted per client IP, and the Gateway honors X-Real-IP only from a peer listed in AUTH_TRUSTED_PROXIES. The compose files never set it, and every browser request reaches the Gateway from the nginx container, so all logins shared nginx's address: five wrong passwords from anyone locked every user out for five minutes. AUTH_TRUSTED_PROXIES now also accepts hostnames, resolved off the event loop and cached for 10 seconds (failed lookups included, and only when X-Real-IP is present), and both compose files default it to the bundled nginx service, whose address Docker assigns. nginx overwrites X-Real-IP with $remote_addr on every Gateway route; a test pins that for the three maintained configs. * docs(changelog): reference #6519 * fix(auth): share one trusted-proxy lookup across concurrent cache misses A burst of setup-status / login requests on an expired hostname entry started one getaddrinfo per request while DNS was slow, occupying the default executor that password verification also uses. The cache check and an in-flight task registration now happen without a yield between them; later callers join that task through asyncio.shield, so a cancelled request cannot cancel the lookup the others joined, and the task caches its own result. The compose default keeps `:-` on purpose: every browser request arrives through nginx, so an empty AUTH_TRUSTED_PROXIES would only bring back the shared lockout. The compose comment and .env.example say so, and the rendered-compose test now covers the explicitly empty value. * fix(docker): keep a .env AUTH_TRUSTED_PROXIES under make docker-start docker.sh start runs Compose from docker/ without --env-file, so the dev compose file's ${AUTH_TRUSTED_PROXIES:-nginx} interpolated from the shell only, and the resolved environment entry outranked the operator's value loaded through env_file. Export it from .env alongside the proxy variables (a shell export still wins), and strip a CRLF before the quotes. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
kuku
|
845af59301
|
fix(models): preserve MiniMax reasoning_content responses (#6071)
* fix(models): preserve MiniMax reasoning_content responses * fix(models): preserve whitespace MiniMax reasoning deltas * test(models): pin MiniMax reasoning precedence --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
b99ac916b6
|
fix(auth): count login lockouts in the database across Gateway replicas (#6501)
* fix(auth): count login lockouts in the database across Gateway replicas POST /api/v1/auth/login/local throttled failed logins per client IP in a module-level dict, so with N Gateway replicas behind one load balancer an attacker got N x auth.local.max_login_attempts guesses and a lockout on one replica was invisible to the others. Move the counter behind a LoginThrottleStore (deerflow.persistence. login_throttle): MemoryLoginThrottleStore keeps the historical per-process semantics and 10k tracked-IP bound; SqlLoginThrottleStore shares one login_throttle row per IP through the application database, incrementing and starting the lock in a single dialect-native upsert, applying check decisions with compare-and-set predicates in a separate write transaction, and sweeping served locks plus 24h-idle counters in bounded batches. Both stores pin the existing contract: the lock starts when the count reaches max_login_attempts, the committed duration is honored when the policy shrinks mid-lock, a served sentence is never resurrected, tightening keeps the count, success resets the IP, and the 429 status and message are unchanged. New startup-only selector auth.local.throttle_storage (auto | memory | db, default auto = the database whenever database.backend is sqlite or postgres). The Gateway installs the store once at startup after the persistence engine; deps._validate_login_throttle_storage warns when a multi-process deployment keeps memory. Migration 0034_login_throttle creates the table; config_version 55 -> 56 in config.example.yaml and the Helm chart. Docs: README, CONFIGURATION.md, AUTH_DESIGN.md, config and migrations AGENTS.md, CHANGELOG. * docs(changelog): link the login throttle entry to #6501 * fix(auth): re-check expiry on the row the login-throttle sweep deletes The bounded cleanup in SqlLoginThrottleStore.record_failure deleted `WHERE ip IN (SELECT ip ... WHERE <served or stale> LIMIT 200)`. On PostgreSQL READ COMMITTED the expiry predicate applied only inside the snapshot-bound subquery: when the DELETE had to wait on a row lock held by a concurrent check() that was extending the same lock under a raised lockout_seconds, PostgreSQL re-evaluated only the outer WHERE on the updated tuple, IP membership still held for the old candidate, and the now-active lock was deleted; the next attempt from that IP got a clean record and reached password verification (review of #6501, P2). Repeat the served/stale predicate on the DELETE target as well (SqlLoginThrottleStore.sweep_statement), keeping the bounded candidate subquery, so the re-evaluation sees the updated row's expiry. The other writes already carry their full compare-and-set predicate on the target row: _discard and the check() UPDATE both use _matches (ip, fail_count, locked_at, lock_duration_seconds), so a tuple changed while they waited no longer matches and the stale decision is skipped. _discard keeps the equality compare-and-set rather than adding an expiry condition: its early-release branch deletes a lock whose committed sentence has not elapsed under the lowered live policy, which an expiry predicate would wrongly block, and the equality predicate already pins every input of the decision. reset stays unconditional by design. Tests: a dialect-compile test asserts the DELETE's own WHERE carries the expiry predicate for postgresql and sqlite (red on the previous HEAD), and a PostgreSQL-only test (skipped without TEST_POSTGRES_URI) reproduces the update-versus-sweep interleaving with two engines, waiting on pg_stat_activity for the sweep to block before the extender commits. Docs: sql.py concurrency contract and the migrations guide's 0034 bullet. * fix(auth): keep an active login lock's clock and fail closed on explicit db Address the "changes requested" review on #6501. Lock clock (confirmed defect): record_failure re-stamped locked_at = now for every failure on an already-locked row, so a sustained attacker's sentence slid forward with each attempt. Both stores now apply one rule in one atomic decision: new_count = fail_count + 1; below max_attempts the IP counts with no lock (a failure after the operator raised the threshold clears an existing lock); at or over it, a failure during an *active* lock keeps locked_at and the committed lock_duration unchanged, while a never-locked or served row starts anew with now / lockout_seconds (so the failure that reaches a raised threshold starts a fresh lock rather than keeping a stale locked_at). The SQL upsert expresses it as nested case() on the row's own values; the memory store mirrors it. The sweeps in both stores now leave the row being recorded to that decision instead of deleting a served lock or stale counter first, which split the contract (SQL would have produced a fresh (1, NULL, NULL) row). base.py states the rule. Explicit db downgrade: resolve_login_throttle_store raises SystemExit — the failure type of the other startup validations in app.gateway.deps — when throttle_storage='db' resolves to the shared table but no engine is initialised, so langgraph_runtime refuses to start instead of silently running every replica on per-process counters; the lazy bare-app path applies the same rule and installs nothing. 'auto' keeps warning and falling back. 'db' on a memory database still degrades with a warning (there is no shared table to insist on). Non-blocking items: _key() logs a warning when a client key is truncated to 255 characters (never the key itself); the sql.py docstring covers wall-clock locked_at across replicas (run NTP; skew shifts evaluation by that much) and why STALE_COUNTER_SECONDS=24h / SWEEP_BATCH_SIZE=200 were chosen; a store-level blocking-I/O anchor covers the SQL reset path; the _validate_login_throttle_storage warning was verified to render 'throttle_storage=memory' for the StrEnum (the gate test now passes the enum member and asserts that text). reset stays unconditional: a peer failure that commits after a reset leaves fail_count=1, one genuine failure after the cleared history, which a new test documents. Docs: CHANGELOG entry, config.example.yaml, CONFIGURATION.md, AUTH_DESIGN.md and the throttle_storage field description. * fix(auth): re-evaluate a login-throttle check whose compare-and-set missed SqlLoginThrottleStore.check applied its decision (commit the live duration, or clear a served / released lock) with a compare-and-set predicate on the snapshot it had read, but ignored the affected-row count. When a peer changed the row between the read and the write — an admitted wrong password incrementing fail_count on another replica, or a successful login clearing the row — the UPDATE or DELETE matched zero rows, yet check reported the decision as applied: it could return an extended 60s sentence while the row kept its 1s duration, so the next check cleared the lock and let password verification through; a live duration decrease could be dropped the same way; and a row a peer had just cleared was still reported as locked (#6501 review, P2). Every compare-and-set write in check now checks result.rowcount. On a miss the row is re-read and re-evaluated from the top, up to CHECK_CAS_ATTEMPTS (3) times, applying a fresh compare-and-set decision on the new snapshot and never clobbering a reset or replacement: a row that is gone is reported as allowed, a fresh peer lock is reported as locked, and an incremented count gets the extension or decrease committed on it. Exhausting the bound fails closed: nothing is written and the answer is the last snapshot's lock under the live policy, logged at debug. The read stays in its own short transaction and each write in a separate one. The evaluation is now one shared pure function, evaluate_check in base.py (CheckDecision), used by both stores so the contract cannot diverge. The memory store keeps a single pass: its compare-and-set cannot miss because there is no await between the read and the write on the loop thread, which its docstring now states. Tests (red first on df26c2acf) use a session proxy that runs the peer mutation on a second engine right after the snapshot read ends its transaction: the reviewer's raised-duration interleaving (59.5s reported and committed, the increment survives, a later check still locked), the mirrored decrease, early release on the fresh row, a reset landing between read and write (0.0, no row recreated), and the bound-exhausted fail-closed case. * docs(changelog): resolve merge markers left in the Unreleased section * test(migrations): pin 0035_login_throttle as the head in the 0034 watermark test Upstream #6477 added test_migration_0034_run_event_seq_watermark.py asserting 0034 is the single head; this branch chains 0035_login_throttle after it, so the test now pins the current head and walks the chain like its siblings. * perf(auth): one session per login-throttle check, one transaction per failure Two non-blocking performance suggestions from the #6501 review. Single session per check. The router probed the row with store.get() and then called store.check(), which re-read it: a recorded IP cost two connection-pool checkouts and two SELECTs per request on the unauthenticated login path, letting a lockout flood compete with legitimate traffic for Postgres connections. LoginThrottleStore.check now takes the policy as an async callable (LoginThrottlePolicy) and resolves it lazily inside the store: the probe, the policy resolution and the compare-and-set loop share one session, a clean IP returns 0.0 without ever calling the policy (the config.yaml read is still skipped for the common case), and a recorded IP resolves it exactly once, also across compare-and-set retries. The decision is made on a snapshot read after the resolution: the policy read is a yield point, and a decision that needs no write has no compare-and-set to catch a row a peer changed meanwhile — so one session, but still a read after the yield. The memory store takes the same signature and re-reads its record after the await for the same reason. The router passes its asyncio.to_thread policy read as the callable and drops the separate probe; get stays on the protocol for tests and anchors. No short-TTL verdict cache: it would mask a successful login's reset for the TTL window, and the single session already halves the hot path. Sweep in the upsert's transaction, plus an index. record_failure ran the housekeeping DELETE in its own transaction and the upsert in a second one, two commits (two WAL flushes on PostgreSQL) per failed login; the sweep now rides in the upsert's transaction with one commit, the keep exclusion already protecting the row being recorded. The new ix_login_throttle_updated_at (ORM __table_args__ and the unreleased 0035 migration, created after the table and dropped before it, each guarded separately so a table left by an interrupted attempt still gets its index) serves the sweep's stale-counter predicate; the served-lock predicate scans a table the sweep itself keeps small. Tests (red first on a137fbc47): a clean IP never resolves the policy and a recorded IP resolves it exactly once (both stores), the SQL store opens exactly one session per check and commits once per record_failure (counting wrappers), the migration test asserts the index after upgrade, after completing a partial upgrade that lacked it, and gone after downgrade while compare_metadata stays empty; the router-level "mutated during policy resolution" tests now inject their mutation through the policy callable. * docs(changelog): note config_version 57 in the login throttle entry |
||
|
Nan Gao
|
5ae953fc2f
|
fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers (#6494)
* fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers
DeerMem's derived SQLite FTS5 index for every user was one WAL database pinned
to {storage_path}/.retrieval. With several Gateway Pods sharing one home volume
each Pod opened that file over a network filesystem (unsupported by SQLite),
every Pod start emptied and refilled it under its peers, one Pod's corruption
recovery deleted files the others held open, and a Pod never learned that a
peer had written a user's facts, so memory_search served stale results.
- New backend-private key memory.backend_config.retrieval_index_path: empty
keeps {storage_path}/.retrieval, a relative path resolves against
storage_path, an absolute path is used as-is. The full rebuild and the
corruption recovery act only on that configured index.
- rebuild_index records each agent scope's manifest signature
(mtime_ns, size, revision) before reading its facts; search_facts rebuilds a
scope whose live signature differs because another process wrote the user's
memory. An own commit advances the recorded signatures of that user's scopes
only when they were in sync at the previous revision, so own writes never
rebuild while an interleaved peer write still does.
- Gateway startup warns, in the style of _validate_agent_storage, when a
declared multi-instance deployment keeps the DeerMem index inside
storage_path.
- config.example.yaml documents the key; config_version 55 -> 56, mirrored in
the Helm chart and its README.
- Helm mounts an emptyDir at /var/lib/deerflow/memory-index, points the key at
it, drops the legacy memory.storage_path: memory.json line, and documents the
memory layout next to the other multi-replica prerequisites.
* docs(changelog): link the memory retrieval index entry to #6494
* fix(memory): never advance the index signature on a no-op commit and resync whole scopes
Two review findings on the cross-replica retrieval re-sync:
- save() and apply_changes() inferred the replaced manifest revision as
"returned revision - 1". The commit helper returns the existing manifest
unchanged for a no-op (unchanged-value or empty change set), so after a peer
bumped the manifest an own no-op write advanced this process's stale index
signature to the peer's revision without carrying the peer's delta: a fact the
peer deleted stayed searchable until the manifest changed again. Both paths
now read the real pre-commit revision under the user lock and pass
committed=(pre_revision, signature) only when the commit produced a new
revision; a no-op passes None so the next search still rebuilds the scope.
- The scoped rebuild (signature mismatch and the pre-existing dirty-scope path)
listed facts through list_facts() with its default limit of 100 and then
replaced every row of the scope, so a scope with more than 100 facts lost its
tail on the first re-sync. It now reads the scope's complete fact list.
* fix(memory): fence index signature promotion against a concurrent scope rebuild
Review finding on the cross-replica re-sync: a successful adapter notification
does not prove its delta is still in the index when the signature is promoted.
The SQLite lock covers single adapter calls only, and neither the scope lock
nor DeerMem's warm-up lock spans "notify, then promote", so rebuild_index can
replace a scope's rows in between. After a peer summary-only commit (r+1), A's
search starts a scoped refresh that captures the r+1 signature and reads f1/f2;
A's own deletion of f2 commits r+2 and removes the row; the refresh then
installs its stale f1/f2 and marks the scope synced at r+1; the deletion
resumes and promoted that signature to r+2, so later searches saw the live
signature as equal and kept returning the resurrected f2 until another manifest
change.
Mechanism: a generation fence rather than a lock. rebuild_index bumps a
per-scope generation (scoped rebuild) or a storage-wide generation (full
rebuild) under _cache_lock whenever it publishes or forgets a scope's rows, in
both mark_synced and forget_synced. _dispatch_retrieval_notifications snapshots
the generations under _cache_lock before its first adapter call and, at
promotion time, skips every scope whose generation moved (and everything when
the storage-wide generation moved), leaving the rebuild's own captured
signature in place; that signature predates the commit, so the next search's
comparison rebuilds the scope. A lock would have had to serialize notification
plus promotion with bulk install plus publication while search_facts ->
rebuild_index -> load() already takes the scope, file and cache locks; the
fence adds no lock-order obligations and only costs a redundant rebuild in the
rare race. Every interleaving is covered: a rebuild that completes before the
snapshot is reflected in the captured generation and promotion proceeds on an
index that carries both the rebuild's rows and this write's delta; one that
publishes between snapshot and promotion is skipped; one that publishes after
the promotion records its stale captured signature, which the next search
catches.
Tests reproduce both orders deterministically with threading.Event ordering
(adapter wrapper that holds the bulk install and the remove notification):
test_promotion_is_fenced_against_a_refresh_that_installs_in_between (red before
the fix) and
test_stale_refresh_installing_after_a_promotion_is_caught_by_the_next_search
(regression guard for the mirrored order).
* fix(memory): publish a rebuild's signature under the lock that replaced its rows
Review finding on the cross-replica re-sync: two peer-triggered refreshes of
one scope could replace rows in one order and publish their captured
signatures in the other. On a warmed Pod A, refresh R1 captures r+1 with f1/f2
and waits before installing; a peer deletes f2 at r+2; refresh R2 captures
r+2, installs f1 only and pauses after bulk_rebuild returns, before
mark_synced; R1 installs its stale f1/f2 and publishes r+1; R2 then publishes
r+2 over R1's rows. Later searches saw the live signature as equal and kept
returning the deleted fact. The generation fence only guards own-write
promotion, and the adapter's SQLite lock ends when the install returns.
Mechanism: a per-storage _retrieval_publish_lock held across the row
replacement and the signature publication, for the bulk path (bulk_rebuild +
mark_synced / forget_synced) and the clear+upsert fallback, scoped and full
alike. Overlapping refreshes of a scope therefore serialize their
install+publish pairs: whichever installs last also publishes last, and a stale
captured signature published last differs from the live manifest, so the next
search's compare rebuilds the scope. Fact reading (load() / the glob) stays
outside the lock so a slow read never blocks refreshes of other scopes. The
startup full rebuild holds the lock for its single bulk install; scoped
refreshes wait for it, which is acceptable because before warm-up completes a
search would be rebuilding its scope anyway (DeerMem.warm_retrieval takes its
own _retrieval_lock only after rebuild() returns, so the two never nest).
Lock order: the publish lock is acquired with no storage lock held --
search_facts releases _cache_lock before calling rebuild_index, reload() leaves
its scope/file-lock block before rebuilding, and DeerMem._ensure_retrieval_scopes
holds only its outer RLock, which nothing inside rebuild_index acquires. Inside
the lock only the adapter's own lock and _cache_lock are taken, and both are
leaf locks that never call back into storage, so no cycle is possible.
Test: test_rebuild_publication_stays_ordered_with_row_replacement drives the
reviewer's interleaving with threading.Event ordering (red before the fix). The
pause harness now holds a refresh after its fact read, outside the publish
lock, instead of inside the adapter's rebuild, so the earlier fence tests keep
their semantics without holding the new lock while paused.
* fix(memory): read each manifest once per full rebuild and serve the local index on a compare read error
Two suggestion-level review items on the cross-replica retrieval re-sync.
- The full rebuild cached manifest signatures keyed by (manifest path, agent
name), but _scope_signature reads only the manifest file -- every agent
bucket of a user shares one memory.json -- so the startup scan stat+read+
parsed the same file once per bucket that has facts. The cache is now keyed
by the manifest path alone while synced signatures are still recorded per
(user, agent) scope: users manifest reads instead of users x agents.
- The per-search freshness compare put a shared-volume manifest read on every
search_facts call, and _load_memory_file raises MemoryStorageCorruption on
OSError and malformed JSON, so a transient network-filesystem read error or a
corrupt memory.json failed a search the Pod-local index could still answer.
The compare now catches OSError, ValueError and MemoryStorageCorruption: it
logs a warning naming the scope (traceback at debug), serves the current
local index for that scope and leaves the recorded synced signature
untouched so the next search compares again. It deliberately does not mark
the scope dirty: a scoped rebuild would hit the same failing read, forget the
scope's signature and fail the search path too.
* docs(changelog): resolve merge markers left in the Unreleased section
* fix(memory): forget a scope's index signature when an own delta cannot be proved compatible
Review finding on the cross-replica re-sync: skipping promotion left a newer
rebuild's signature trusted even after this dispatcher had overwritten its rows
with an older notification. On a warmed Pod A, an own PATCH of f2 commits r+1
and pauses after releasing the user lock, before its adapter upsert; Pod B
deletes f2 at r+2; a concurrent search on A detects the peer change, rebuilds
the scope without f2 and publishes r+2; the pending r+1 upsert then reinserts
the deleted fact, and the revision check merely skipped because the synced
revision was r+2 rather than the PATCH's previous r. The signature stayed equal
to the live manifest, so searches returned the deleted fact indefinitely. The
generation fence could not help: it only suppresses promotion, the stale
adapter mutation still happens.
Rule: after an own commit's notifications are applied, the mutated scope's
recorded signature may stay only when the delta is provably compatible with
the published snapshot -- the scope is synced at exactly previous_revision and
no rebuild published rows for it (or storage-wide) since the fence snapshot --
in which case it is promoted as before. In every other case where this
dispatcher mutated the adapter for the scope (synced at another revision,
newer or older; generation moved; not synced at all; or notifications without
a committed revision, i.e. migration and clear paths) the scope's signature is
forgotten so the next search's compare rebuilds it. The user's other scopes
were not mutated by this delta: they are promoted when provable and otherwise
left untouched. The decision runs under _cache_lock together with the
generation reads, so a rebuild publishing concurrently is either observed or
observes the forget; the failure path (dirty + forget) is unchanged.
Orders covered: the reviewer's (publish r+2, then the stale upsert) forgets and
the next search rebuilds without f2; the earlier fence order (publish between
adapter op and promotion) now forgets too, one redundant rebuild; no concurrent
rebuild promotes exactly as before.
Test: test_stale_own_delta_replayed_over_a_newer_snapshot_forgets_the_scope
holds the PATCH's adapter upsert after the user lock is released (new
pause_next_upsert hook on the test adapter), Event-ordered, red before the fix.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
litescraper
|
e1458eefee
|
feat: support configurable Serper base URL (#6483)
* feat: support configurable Serper base URL * fix(serper): address configurable endpoint review feedback * fix(serper): address endpoint configuration review * fix(serper): pair provider settings and validate endpoints * test(serper): account for URL redaction in endpoint diagnostics --------- Co-authored-by: litescraper <322634161+litescraper@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
ZJPex
|
220d62d79a
|
feat(memory): add scoped fact lookup by ID (#6443)
* feat(memory): add scoped fact lookup by ID * docs(memory): synchronize tool-mode lookup guidance * docs(memory): clarify named-agent read capability requirement |