225 Commits

Author	SHA1	Message	Date
Hyeonsang Cho	9146bfa03d	feature(gateway): issue request trace ids unconditionally (#5119) ... * refactor(gateway): issue request trace ids unconditionally The request trace id was gated behind logging.enhance.enabled at every entry point, so downstream code had to keep asking whether one existed: a header-provenance flag in its own ContextVar, a precedence resolver, and three-level carrier fallbacks at each consumer. Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP; ensure_trace_context covers the entry points that never touch ASGI -- scheduled occurrences, MCP task notification runs, IM channel messages, and the embedded client -- each scoped to one unit of work so a long-lived worker task cannot leak one occurrence's id into the next. The ContextVar becomes the only source; the response header, runtime context, run metadata and log records are derived outputs. Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and drop their presence guards. Removed: resolve_deerflow_trace_id, the header-provenance flag and its three helpers, set/reset_current_trace_id, is_trace_correlation_enabled and its gateway alias. BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and it cannot be turned off; logging.enhance.enabled controls log output only. Installations on the default enabled: false will start seeing the header. No config keys were added or removed. * fix(gateway): stop persisting a caller-supplied trace id on the run record body.metadata forks two ways: through build_run_config into the live run config, which the run worker restamps, and through create_or_reject into the run record that the runs API echoes verbatim. Only the first was covered, so a client sending metadata.deerflow_trace_id made the most durable and most visible surface of a run disagree with the X-Trace-Id and the log lines the same request produced -- a correlation id that does not match the logs is worse than none. Stamp the server-issued id once at the trust boundary so both forks receive it, preserving the caller's own metadata keys. Close the same gap on config.context, which reaches the runtime context by a separate path: _build_runtime_context no longer merges server-owned keys from the caller, and _install_runtime_context assigns rather than setdefaults. A thread's metadata is no longer seeded with the run-scoped id of whichever run created it -- one thread spans many runs and as many trace ids. Found by driving a real run through the Gateway and reading the run back from the runs API; every unit test built its metadata by hand and so could not see it. * fix(gateway): expose X-Trace-Id to split-origin browser clients X-Trace-Id is not on the CORS safelist, so a browser client served from a separate origin could not read it -- and those are exactly the clients that cannot read the Gateway's logs either, leaving them with nothing to quote in a bug report. Same-origin nginx deployments were unaffected, which is why this stayed hidden. Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing TRACE_ID_HEADER rather than repeating the literal. * fix(gateway): keep X-Trace-Id on unhandled-exception 500s Starlette's ServerErrorMiddleware sits outside every user middleware and emits unhandled-exception 500s through the raw send, so those responses never pass TraceMiddleware's header-writing wrapper. The 500 for a server bug is exactly the response a user most needs to correlate with a log line, and it was the one response that shipped without the id. TraceMiddleware now tracks whether http.response.start has been sent. On an exception with no response started it emits its own plain 500 carrying the header, then re-raises: the outer ServerErrorMiddleware sees the response already started and only re-raises too, so the server's exception logging is untouched. An exception mid-stream keeps propagating unchanged — a second response start cannot be sent, and the already-written header stands. The trace id is printable ASCII by construction (normalize_trace_id / generate_trace_id), which is what makes the raw latin-1 header encoding safe. * fix(gateway): strip the forged trace id from the persisted request echo The run-record fix stopped a forged metadata.deerflow_trace_id on the authoritative metadata surface, but the raw request echo still carried one: create_or_reject persists body.config verbatim as runs.kwargs_json, which the runs API serves back. A client posting config.context.deerflow_trace_id therefore still got its forged value stored and echoed on one API surface while the header, logs, run metadata, and checkpoint all carried the real id — the id is ignored as input there, so echoing it back only manufactures disagreement. Two changes close it. redact_config_secrets — already the shared scrub for that echo, applied at admission and again at serve time, so historical records are covered too — now also drops deerflow_trace_id from config.metadata and config.context. And build_run_config now merges run metadata onto a copy of the caller's config["metadata"] instead of updating it in place: the nested values of the request config are reference copies, so the in-place merge was writing the server-stamped key through into body.config, contaminating the "what the client sent" record before it was persisted (and incidentally masking the forged-value echo on the metadata container). The regression test posts a forged id through body.metadata, config.metadata, and config.context at once and reads the kwargs echo back off the run record, failing if either leak returns. * docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence The trace section of the harness AGENTS.md now covers the two fixes that close the derived-output rule (the kwargs-echo scrub in redact_config_secrets plus build_run_config's copy merge, and TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains their Fixed entries. It also writes down the one accepted divergence: a crash-recovered scheduled launch reuses the durable run through its idempotency key, and start_run returns early on idempotency_reused without restamping — so the run record keeps the first attempt's deerflow_trace_id while the retry's own log lines carry the freshly minted id of its ensure_trace_context binding. The divergence is confined to the crash-recovery window and is accepted rather than fixed: restamping on reuse would rewrite a persisted record for a run that already exists, which is worse than two ids that each correlate their own attempt's logs. Written down so the next reader of the scheduler recovery path does not diagnose it as a bug. * docs(config): align the logging.enhance schema note with the unconditional trace id The config-module AGENTS.md still described logging.enhance as the gate for the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is gone: ids are issued unconditionally and this block decides log output only. Left as-is, the stale wording invites an agent to "restore" a header gate it believes was lost. Reworded to match the sibling AGENTS.md files and config.example.yaml, with a pointer to the Request Trace Context section that owns the full model. * docs(changelog): link the trace entries to #5119 The five new entries pointed at the ([#XXXX]) placeholder with no reference definition, rendering as literal text instead of a link — and RELEASING.md step 2 relies on those references when the section becomes release notes. All five now point at #5119, with the definition appended to the reference block. * refactor(harness): rename _stream_without_trace_context to _stream_turn The name asserted the opposite of what the method now does. It was accurate while logging.enhance.enabled could route stream() around the trace scope; with the gate gone it is the only stream implementation left, and it binds the id itself via ensure_trace_id(). Private, so the rename touches only the definition and the one stream() call site. * docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget The expanded Request Trace Context section pushed the effective AGENTS.md chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the section from 7,359 to 4592 bytes with no facts removed: the entry-point table, the derived-output rule and its enforcement points, the accepted scheduled-retry divergence, the two resolution helpers, the stream() binding rationale, the log-output-only gate, the CORS listing, the 500 fallback, and the test map all remain. Sized against the merge, not just the branch: current main grew the same chain by ~724 bytes, so the check was verified on the merged tree as well (97,772 bytes; branch tree 97,048). * fix(gateway): declare content-length on the fallback 500 The pre-response 500 declared content-type but no content-length, leaving the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response it replaces, which sends content-length: 21. The explicit header keeps the fallback byte-identical to what clients saw before. * docs(readme): drop the trace-correlation condition from the translations The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id matches X-Trace-Id "when request trace correlation is enabled". The id now always matches and that condition no longer exists, so each bullet states the unconditional match and that logging.enhance.enabled only controls whether the id is printed into logs — the one piece of the feature a user can still configure. * test(gateway): pin TraceMiddleware wiring through create_app() Every X-Trace-Id test exercised a hand-built four-route app, so the real stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting it — or short-circuiting above it — passed CI while silently dropping both the response header and the ambient id the run-record stamp and enhanced log records derive from. One case now drives /health through create_app() and asserts the inbound id round-trips; mutation-checked by removing the wiring line, which fails exactly this test. * docs(gateway): note the fallback 500 is CORS-opaque The pre-response 500 is emitted outside CORSMiddleware — the exception has already unwound past it — so it carries no Access-Control-Allow-Origin and a split-origin browser client cannot read the id on this one response, unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the class and in the CHANGELOG entry rather than fixed: replicating the origin allowlist outside CORSMiddleware would let the two policies drift. * fix(harness): keep abandoned-stream cleanup inside the trace binding stream() binds the turn's id around each next(inner) and resets it before yielding, but the finally's inner.close() ran after that binding was gone. Abandoning the stream therefore drove the inner LangGraph generator's GeneratorExit/finally path with no trace id — or an unrelated ambient one from whichever context ran the close — so cancellation and finalization logs and callbacks did not correlate with the turn they belong to. inner.close() is now wrapped in a local bind/reset of the same turn id. The token is set and reset in the same frame, never across a yield, so the per-step cross-context safety is preserved even when GC closes the generator from another Context — pinned by the existing copy_context close test, which now exercises this path. The regression test records the id from the inner generator's finally and fails without the binding. * test(harness): teach the worker-trace fake about RunManager.cleanup Upstream #5112 (bound gateway memory after terminal runs) added a run_manager.cleanup(run_id) call to run_agent's finalization, so the merge-commit CI run failed all five worker-trace-binding tests with AttributeError on this PR's _FakeRunManager. The fake gains the same no-op shape as its other methods. * docs(gateway): bring the gateway AGENTS.md back under its soft budget Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over the 40,960 soft budget that test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes enforces — its Unit Tests run on main was cancelled by push concurrency, so main is currently red on that test and every PR merge-run inherits the failure. Two whitespace/wording trims in the row #5092 touched (a doubled space, and "its configured `context_window`" → "its `context_window`") bring the file to 40,953 with no content change. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-01 16:49:39 +08:00
Aari	317577e285	fix: enforce custom agent skill allowlists in sandboxes (#5077) ... * fix: enforce agent skill allowlists in sandboxes * fix: guard E2B skill projection resets * fix: preserve agent skill isolation across delegation * fix: close sandbox skill isolation bypasses * fix(sandbox): close skill isolation review gaps * fix(sandbox): harden skill isolation lifecycle	2026-08-31 14:35:08 +08:00
Wuong	e12925458a	feat(streaming): make heartbeat interval configurable (#5017) ... Co-authored-by: Wuong <26929475+Wuong@users.noreply.github.com>	2026-08-30 10:46:01 +08:00
Aniket Wagh	8a830f6354	fix(deps): depend on renamed tenki package instead of tenki-sandbox (#5087) ... * fix(deps): depend on renamed tenki package instead of tenki-sandbox tenki-sandbox has been removed from PyPI and republished as tenki. Its old wheel URL still resolves, so existing lockfiles keep installing and the breakage is invisible to anyone with a warm lock; any fresh resolution fails with 'tenki-sandbox was not found in the package registry'. tenki 1.0.2 still ships the tenki_sandbox module, so the imports in community/tenki/provider.py and sandbox.py are unchanged. Fixes #5081 * fix(tenki): point install guidance at the renamed distribution The rename to `tenki` left the user-facing remediation still naming the removed package. `_import_client` raised "pip install tenki-sandbox" on the missing-extra path — the exact instruction this change proves now 404s on PyPI, handed to the user at the exact moment they need it to work. Update that message and the remaining `tenki-sandbox` references in the provider, sandbox adapter, README, sandbox AGENTS.md and the test docstring. The imported module stays `tenki_sandbox`, so the distribution and module names now differ; each mention says so rather than just swapping the string. No behavior change beyond the error text. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tenki): migrate the provider to the 1.x workspace-only API Renaming the dependency was not enough. tenki 1.0.2 keeps the tenki_sandbox module name but not its contract: Client.create dropped project_id and has no **kwargs to absorb it, and IdentityWorkspace no longer carries `projects` (the attribute is gone from the package entirely). Both configuration paths therefore failed before a sandbox could be created — explicit project scope raised TypeError, and automatic scope raised AttributeError walking workspace.projects. Scope is now the workspace alone. _resolve_scope returns a single workspace id, auto-selecting when the account has exactly one, and project_id is gone from create_kwargs and from the documented config surface. A stale project_id in config.yaml warns rather than fails. SandboxConfig is extra="allow", so simply not reading the key would leave it scoping nothing with no signal; it also used to short-circuit the identity lookup, so operators with more than one workspace need to know they must now set workspace_id. The suite passed against the broken provider because the fake client took **kwargs and swallowed the project_id the real SDK rejects. The double now mirrors 1.0.2 — keyword-only, no **kwargs — so an unexpected argument is a TypeError in tests exactly as it is against the SDK. Reintroducing the old create call fails 20 tests; before this change it failed none. Verified against the exact locked wheels: every other kwarg the provider passes (name, workspace_id, sticky, wait, max_duration, image, cpu_cores, memory_mb, env) and every SDK surface it touches (who_am_i, Identity.workspaces, wait_ready, exec, close, the fs API, the four terminal exception classes) is unchanged in 1.0.2. Reported by willem-bd in review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(config): drop sandbox.project_id from the Tenki example The canonical example still documented project_id as a supported optional key after the provider stopped honouring it, so an operator following it could set the key, get no scope from it, and hit a workspace-resolution failure with nothing in the example to explain why. Replaced with a migration note rather than a silent deletion: someone upgrading already has the key in their config.yaml and needs to know it is inert now and that workspace_id is what scopes a sandbox on Tenki 1.x. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Aniket Wagh <aniketwaghh@users.noreply.github.com>	2026-08-30 10:37:58 +08:00
Nefelibata	c6f6a01f56	fix(runtime): prevent IndexError in MemoryStreamBridge._make_gap on empty events buffer (#5047) ... * fix(runtime): prevent IndexError in MemoryStreamBridge._make_gap on empty events buffer * fix(runtime): handle empty stream replay gap bounds across backend and frontend - Clamp MemoryStreamBridge queue_maxsize at 1 and validate StreamBridgeConfig.queue_maxsize >= 1 - Update StreamGap docstring to clarify None retained bounds - Allow StreamReplayGapData and parseStreamReplayGap in frontend to accept string | null bounds, safely resuming when bounds are null - Add backend and frontend regression unit tests for queue clamping and null bounds replay gap * docs(stream-bridge): bump config_version and document empty buffer replay gap behavior * docs: document nullable gap bounds and sync helm config_version to 37	2026-08-29 17:21:29 +08:00
Nan Gao	bf3e792a6a	feat(models): add GLM-5.3-Flash thinking workaround (#5074)	2026-08-28 22:24:10 +08:00
Serply	4dbfe37ff3	feat(community): add Serply web search tool (#5023) ... Add deerflow.community.serply.tools:web_search_tool, a Google SERP provider for the web_search slot that also covers Google News and Google Scholar through an optional `vertical` config option. Reads the key from api_key in config.yaml or SERPLY_API_KEY, clamps max_results to Serply's 1-100 range, and returns the same structured JSON errors as the Serper and Brave tools. Register the provider in config.example.yaml, scripts/doctor.py, scripts/wizard/providers.py, .env.example, backend/docs/CONFIGURATION.md, the en/zh tools.mdx provider tabs, and tools/AGENTS.md. Tests mock httpx.	2026-08-28 10:30:30 +08:00
yong	846c716523	feat(search): add Tencent Cloud WSA provider (#5057) ... * feat(search): add Tencent Cloud WSA provider * docs: restore README to upstream * docs: remove README changes from WSA provider PR * fix(doctor): validate Tencent WSA API key	2026-08-27 18:12:05 +08:00
zhangwei139623	431892e16a	feat(knowledge): add read-only RAGFlow retrieval (#4955) ... * feat(knowledge): add read-only RAGFlow retrieval * test(knowledge): cover RAGFlow retrieval contracts * docs(knowledge): document retrieval-only RAGFlow setup * refactor(knowledge): move RAGFlow settings to tool config * fix(ragflow): bind retrieval to configured datasets * docs(ragflow): record validated response versions * fix(ragflow): bind retrieval by dataset id * fix(ragflow): search all datasets by default * fix(ragflow): retrieve mixed embeddings by group * docs(ragflow): keep feature details out of agent guides * docs(ragflow): remove agent guide changes * docs(ragflow): remove root readme changes * fix(ragflow): handle unresolved and empty datasets * fix(ragflow): harden dataset scope and errors	2026-08-25 16:57:55 +08:00
Aari	ff0a6768c2	feat(subagents): add unified capacity and durable batch execution (#4998) ... * feat(subagents): add capacity controls and durable batches * fix(helm): sync subagent config schema version * fix(subagents): preserve batch history without worker * fix(subagents): support explicit factory runtimes * fix: address durable batch review findings	2026-08-25 07:49:38 +08:00
hataa	cc6a2657e7	feat(authz): enforce sandbox:execute authorization at sandbox acquisition (#4063 Phase 3) (#4911) ... Sandbox is an execution environment, not a named resource: multiple tools (bash, read_file, write_file, glob, grep, ...) depend on it, all funneled through ensure_sandbox_initialized / ensure_sandbox_initialized_async. Gate the single acquisition entry point (single source of truth) instead of maintaining a sandbox-tool-name set in middleware: - authorize_sandbox_execution helper (authz/sandbox_authz.py) checks authorize("sandbox", "execute", target="*") — a binary judgment (can this role use the sandbox at all); RBAC allow:"*"/true permits, allow:[]/false denies. - lazy path: ensure_sandbox_initialized (+ async) calls the gate before provider.acquire. - eager path: SandboxMiddleware.before_agent / abefore_agent call the gate before _acquire_sandbox. - deny raises SandboxAuthorizationError (SandboxError subclass) which propagates through tool execution as a friendly ToolMessage (RFC §9: 'not a crash'). - authorization.enabled: false is a no-op everywhere; provider errors follow fail_closed (deny) / fail_open (allow). 12 tests in tests/test_sandbox_authorization.py cover disabled/allow/deny/ deny-via-bool/no-policy-unrestricted/provider-error-fail-closed/open/ internal-caller + ensure_sandbox_initialized deny (never acquires) and allow (acquires) integration paths.	2026-08-24 16:32:06 +08:00
Aari	645ca08f16	fix(scheduler): enqueue busy scheduled task runs (#4918) ... * fix(frontend): clarify reuse-thread scheduling behavior * fix(scheduler): enqueue overlapping scheduled runs * fix(scheduler): preserve queue lease fencing * fix(scheduler): close queue concurrency races * fix(scheduler): harden queue timeout bookkeeping * fix(scheduler): preserve manual failure schedule --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-08-24 15:38:32 +08:00
Aari	1aa813ddb3	feat: add managed subagents and delegation scopes (#4887) ... * feat: manage and scope subagents * fix: address subagent review feedback * fix: address managed subagent review feedback * fix: harden subagent settings semantics * fix: harden managed subagent cache invalidation * fix: reuse assembled lead agent inputs * fix: migrate managed subagent definitions --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-08-24 11:04:23 +08:00
Aleksandr Sapronov	613b90b0e6	feat(scheduler): make scheduled-run recursion_limit configurable (#4848) ... Adds scheduler.recursion_limit to config.yaml (default 1000, clamped by max_recursion_limit) so scheduled background runs can use a different recursion limit than the web UI. The value is read at dispatch time, so a YAML edit applies to the next scheduled run without a Gateway restart. Also logs a warning when the resolver falls back to the default or clamps the configured value.	2026-08-24 07:09:08 +08:00
muguo	917fe595fc	feat(sandbox): add OpenSandbox provider (#4877)	2026-08-23 15:46:10 +08:00
Zeren Wang	4e35f0d1d4	feat(harness): deterministic tool receipts with model-visible ledger (RFC #4651, layer 1) (#4659) ... * feat(harness): add deterministic tool receipts with model-visible ledger Stamp an immutable per-call fact record (tool name, status, args/output hashes, byte count, timestamp) onto every tool result via a new ToolReceiptMiddleware, and inject the derived receipt ledger (r1..rN) into the model context so subagent reports can cite executed actions. - tool_receipt.py: receipt core (make/extract/render), newest-first budget eviction, ids derived from the append-only message stream - ToolReceiptMiddleware: stamps ToolMessages directly or inside Command-wrapped results; hidden ledger injection mirrors DurableContextMiddleware; sits between ToolProgress and ToolErrorHandling with a build-time ordering guard - config: new verification section (receipts on, judge off), config version 32 -> 33 with example/helm/docs updates * feat(harness): split receipt rendering from stamping; address PR review Review fixes (PR #4659): - output_sha256 now uses sort_keys=True for structured content, matching the order-invariant args fingerprint - stamping failures log at warning (silent ledger gaps would corrupt citations); tool execution remains never blocked - _insert_after_leading_system_messages extracted to shared public message_utils.insert_after_leading_system_messages; both middlewares depend on it instead of a private cross-module helper - code comments in English RFC #4651 revision-2 alignment: - receipts_render_mode config ('always' | 'delegation_only'): subagent chains always render the ledger (citations are produced there); the lead chain renders only while processing subagent results, removing the always-on token tax from ordinary turns - receipts gain bounded args_preview/output_preview (<=200 chars, tail for output) so later typed claim bindings (tests_passed) can anchor to a specific recorded execution * docs(harness): state receipt freshness caveat and vocabulary layering in module docstring * merge: upstream/main — resolve AGENTS.md split, bump config_version to 34, drop unused receipt previews - backend/AGENTS.md: take upstream's slimmed root guidance (#4799); move the ToolReceiptMiddleware chain entry into agents/middlewares/AGENTS.md and the verification.* hot-reload mention into config/AGENTS.md - config.example.yaml + helm values/README: config_version 33 -> 34 so existing v33 configs get the outdated-config prompt (review: willem-bd) - tool_receipt.py: drop args_preview/output_preview — no Layer 1 consumer reads them; re-add with the Layer 2 claim-binding consumer (review: willem-bd) * docs(harness): cover receipt id renumbering after compaction in module docstring Positional display ids are stable only while history is append-only; compaction drops ToolMessages and the survivors renumber, so Layer 2 citation verification must resolve [rN] against the ledger as of the citing turn (review: willem-bd, doc-only). * chore(config): bump config_version to 35 main reached 34 via #4780 without the verification section; publishing the new schema at the same number would silently skip the outdated-config prompt for configs synced from main in that window (review: willem-bd). * fix(skills): restore errno import dropped upstream in #4830 upstream/main adf6c422 uses errno.ENOTDIR in the drift guard but removed the import, so the PR merge ref fails lint-backend (F821). * fix(harness): harden tool receipts against forgery and turn-scope delegation_only Address willem-bd's pre-merge review on #4659: 1. Untrusted receipt metadata: the gateway now strips the server-owned deerflow_tool_receipt key from external input messages; stamping always overwrites any tool-supplied value instead of preserving it; and extract_tool_receipts validates persisted receipt shapes (required typed fields, unknown keys ignored) so malformed entries are skipped instead of crashing render or passing as runtime-stamped evidence. 2. delegation_only no longer sticks on: _should_render now scopes the subagent_status scan to the current turn (messages after the latest genuine user message), so an old completed delegation stops rendering the ledger on later ordinary turns. The genuine-user predicate moves to message_utils.is_genuine_user_message, shared with input sanitization. * fix(harness): stamp receipts outside short-circuiting tool middlewares Address willem-bd's review on #4659: ToolReceiptMiddleware was registered inside Guardrail/SandboxAudit/ReadBeforeWrite/ToolProgress, each of which can return a ToolMessage without invoking its handler — blocked calls (e.g. a read-before-write-denied write_file) never got a receipt, silently gapping the ledger on a default-enabled path. SandboxAudit additionally rebuilds medium-risk results, dropping an inner stamp. ToolReceiptMiddleware is now the outermost wrap_tool_call layer in the runtime tail. Normal results still carry deerflow_tool_meta (stamped by ToolErrorHandling on the inner return path); short-circuit messages self-stamp meta or fall back to message.status. The new invariant is declared as ordering constraints in deerflow.extensions.ordering, with composed-chain regression tests for a blocked write and a warn-rebuilt bash result.	2026-08-23 15:43:37 +08:00
Nan Gao	13f0a7f263	feat(extensions): let an out-of-tree extension observe what the agent did (#4863) ... * feat(extensions): let an out-of-tree extension observe what the agent did DeerFlow's extension system can contribute middleware, services and routes, but an extension cannot answer basic questions about a run without reaching into host internals. Several of the facts it would need are destroyed by the operations that produce them: * The middleware chain injects and rewrites a lot of context — date reminders, recalled memory, compaction summaries, durable-context data, image payloads, activated skill bodies. Downstream, none of it is attributable: at the model-call boundary an injected HumanMessage is indistinguishable from the user's own, and anything wanting to tell them apart has to pattern-match prompt wording, which breaks on the next copy edit. * Two runs of "the same agent" are only comparable if the chain enforced the same limits, prompts and thresholds. Recovering that from outside means reading private attributes and guessing which of them change behaviour — a guess that rots silently as middlewares gain fields. * The lead-agent factory resolves a model after runtime overrides, renders a prompt, filters tools through authorization and composes a stack, all inside one synchronous call, and none of it survives: a middleware sees its neighbours but not the prompt, the run worker sees a graph but not what went into it. * Summarization is destructive by design. N messages leave the context and one summary enters it; afterwards only the summary exists, so "which messages became this?" is not reconstructible. This adds seven neutral facilities so those facts are recorded where they are still true, and releases the contract package as 0.2.0. Message provenance Producers stamp `deerflow_content_kind` / `deerflow_producer_kind` onto the messages they inject or rewrite. Stamping is unconditional — a fact whose presence depends on whether an observer is installed is not a fact — and the keys are server-owned, so provenance cannot be forged from a request. Middleware self-description Twelve middlewares declare their own behaviour-affecting parameters through a duck-typed `release_policy_parameters()`. Long text is hashed rather than embedded: a declaration is an identity, not a copy of the prompt. Agent assembly descriptor `assemble_lead_agent()` returns the graph plus a descriptor whose fingerprint answers "did anything about this agent change between these two runs?". `make_lead_agent()` keeps its graph-only signature — it is the LangGraph Server ABI declared in langgraph.json. Tools and skills are sorted before hashing because their assembly order is incidental; middlewares are not, because stack order decides what wraps what. Host build identity is reported but excluded from the fingerprint, so a redeploy does not invalidate every agent's identity. Context compaction observation Summarization emits the content hashes of the messages it is about to remove joined to the summary that replaced them. Content is the only identity available at that seam: the summary does not become a message, and what later projects it into a request renders it bounded and escaped rather than verbatim. Neutral policy, transform and MCP-source facts Guardrail decisions are published to runtime context under a `__`-prefixed key; result-rewriting middlewares append a declared, ordered transform trail; MCP tools carry their credential-free logical origin. Extension route identity Contributed routes are session-authenticated and cannot opt out, but "logged in" and "administrator" are different questions. Extensions get a neutral projection of the caller rather than the host's auth context, and `require_admin` fails closed when identity cannot be determined. Extension-owned tables An extension that persists data owns its own MetaData and migration chain, so its tables are absent from Base.metadata and `alembic revision --autogenerate` proposes dropping them. Extensions declare a table prefix, which is rejected at registration if it would shadow a host table. The contract package stays dependency-free and imports no host code; every new Protocol method has a default so later additions remain additive. The loader's pre-1.0 rule requires an exact major.minor match, so extensions written against 0.1 are now refused at startup with an actionable install hint rather than loading into a host that implements a different surface. uv.lock records the contract package's new version, so `uv sync --locked` still resolves on a fresh checkout. * fix(backend): sort gateway service imports	2026-08-23 09:57:12 +08:00
Aari	5ffaa09f5a	feat(memory): add hybrid fact eviction policy (#4789) ... * feat(memory): add hybrid fact eviction policy * refactor(memory): simplify confirmation count update * fix(memory): clean up eviction audit metadata * fix(memory): harden eviction cleanup boundaries * fix(memory): address hybrid eviction review	2026-08-17 08:20:42 +08:00
DanielWalnut	062ba9ddfc	feat: integrate MiniMax Code as a native ACP agent (#4846) ... * feat: integrate MiniMax Code as an ACP agent * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>	2026-08-17 08:16:14 +08:00
Aari	47b258ebd7	feat(mcp): add ordinary durable task driver (#4690) ... * feat(mcp): add durable task runtime foundation * fix(chart): sync embedded config version * fix(mcp): isolate task polls during shutdown * feat(mcp): track consecutive poll errors on mcp_tasks poll_attempt_count grows on every claim (successful polls included), so it cannot drive a failure backoff without misjudging normal long tasks. Add consecutive_poll_error_count: incremented when a claim is released after a poll error, reset to zero by any applied snapshot. The backoff/terminal policy that consumes it lands with the first concrete driver. * fix(mcp): harden durable task lifecycle * feat(mcp): add ordinary durable task driver * test(mcp): address durable task review feedback * fix(mcp): preserve submit tool descriptions * fix(mcp): bound remote task calls * fix(mcp): bound persisted task payloads * fix(mcp): preserve task tool error details * fix(mcp): enforce durable task boundaries * test(mcp): cover task config snapshot lifecycle --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-08-15 14:26:38 +08:00
rain02333z-spec	828363705a	fix(scheduler): support safe multi-instance scheduler recovery (#4713) ... * fix(scheduler): reject unsafe multi-worker startup * fix(scheduler): support safe multi-instance recovery * fix(scheduler): make multi-instance recovery lease-safe * fix(scheduler): address multi-instance review feedback * docs(scheduler): add multi-instance upgrade notes --------- Co-authored-by: rain02333z-spec <225106191+rain02333z-spec@users.noreply.github.com>	2026-08-14 23:50:40 +08:00
Eilen Shin	ce4ef1bb2f	fix(channels): bound inbound intake and worker lifecycle (#4800) ... * fix(channels): bound inbound intake and worker lifecycle * fix(channels): harden overload retry and shutdown draining * fix(channels): retain shutdown task ownership	2026-08-14 12:35:24 +08:00
Nan Gao	c542185a7f	feat(extensions): add gateway contribution points and packaged extension management (#4780) ... * feat(extensions): add gateway services and routers * feat(extensions): add standalone reference extension * fix(extensions): harden contributed gateway routes * docs(extensions): document gateway contribution points * feat(extensions): add operator CLI for packaged extension management Add `deerflow extensions install/list/enable/disable/remove` plus the root `make extension-*` wrappers, backed by an `ExtensionManager` that owns one transaction over backend/pyproject.toml, backend/uv.lock, the managed source snapshot, the uv environment, and the `plugins:` block in config.yaml. Install accepts a package requirement, a public HTTPS Git URL, or a local directory. Local directories are copied to backend/extensions/sources/ as deployable snapshots rather than editable installs, and the root .dockerignore re-includes that tree so snapshots reach the backend builder. Remote sources are HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock Docker builder cannot reproduce them. Because environment configuration can still resolve a plain package name to a local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed by an audit of the new lock: any local reference the stock image build cannot reproduce rolls back the whole transaction. A config carrying duplicate top-level `plugins:` keys is rejected outright rather than managed against one block while the Gateway reads another. Dependency synchronization now has one lock authority. The `extensions` dependency group joins [tool.uv].default-groups, every startup path syncs the same lock with --locked and launches with --no-sync, and the Docker images move to uv 0.11.1 for the --no-workspace boundary the manager needs. Loader gains `enabled`, `name` and `package` fields so a disabled extension is skipped before resolution and import. Co-authored-by: Codex <codex@openai.com> * fix(extensions): stop the managed plugins rewrite from destroying config Two data-safety defects in the managed `plugins:` block writer. The "next top-level key" boundary was a regex matching only `[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a config may legally carry any top-level key, and a key the pattern cannot recognize did not fail loudly — it read as "no next section", and the rewrite replaced that neighbour and its entire subtree with the managed block. `my.key`, `2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a plain `extension-enable`/`disable`. Both boundaries now come from the YAML parser's node marks, so key shape is irrelevant. The file-final branch never consulted the trailing-comment scan the has-next-key branch used, so any comment below the block was dropped. Since the manager appends `plugins:` at end of file, that is the steady-state shape for most installs: an operator note below the block was destroyed on the next toggle. Separately, every managed install wrote `required: true` while the loader defaults to false. That turned any later load failure — broken wheel, missing native library, deleted snapshot — into a Gateway startup abort recoverable only with shell access. New records are now written `required: false`, with an explicit `install --required` opt-in; adopting an existing hand-written record still preserves the operator's own choice. * fix(extensions): harden the manager transaction and correct its docs Follow-up hardening on the extension package manager. Security posture, which the docs already claimed: - Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and `UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps the interpreter that the entry-point probe then imports and calls, and every later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS validation the HTTPS-only source rule depends on. Neither is an index, proxy, cache or credential-provider setting, so neither was covered by the carve-out. - Recognize run-together and all-caps secret query parameters (`accesstoken`, `ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires on case transitions, so only the separated spellings were caught. Short generic words stay boundary-anchored, so `?keyword=` remains installable. - Validate the config before running any uv command. `uv add`/`uv sync` execute the package's build backend, so a config the manager could never write to must fail before that code runs rather than afterwards through rollback. Transaction integrity: - Run the second dependency-file restore from a `finally`. The recovery sync runs without `--locked` when the checkout had no lock, so uv writes one while resolving; if that sync then failed, the restore was skipped and the operator kept a lock file they never had. A failing recovery sync now also reports the original failure instead of replacing it. - Skip the recovery sync on cancellation. Answering Ctrl-C with a full dependency resolve invites a second interrupt that escapes the handler and strands the checkout mid-transaction; the declarations are already restored and the next locked startup sync reconciles the environment. - Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so contention surfaced as `Permission denied` rather than serializing. - Locate the entry-point probe's JSON payload instead of parsing stdout's first line, so a `sitecustomize`/`.pth` banner cannot roll back a good install. - Warn when the lock records a loopback source. `127.0.0.1` inside the image builder is a different machine, but unlike an environment-driven wheelhouse resolution this is a source the operator typed deliberately, so it is reported rather than rolled back. Private-network indexes are untouched: a builder on that network can reach them. Docs: the blanket claim that failed operations restore the config file was wrong — the conflict branches deliberately preserve a concurrent external edit and leave `remove` deactivated. Document that, the `required: false` default, the config preflight, the interrupt behaviour, and where the plugins-block boundaries come from. * test(gateway): pin the request-path projection agreement `get_request_route_path()` imports the private `starlette._utils.get_route_path` so the auth and CSRF predicates classify the exact string Starlette's router matches on. Its requirement is not "strip root_path correctly" but "return what the dispatcher is matching", so delegating to the router's own implementation keeps the two in lockstep by construction. Keep the private import rather than vendoring a copy: an import that disappears fails loudly at startup, while a stale copy diverges silently at a security boundary. Cover the property directly instead of the mechanism, so the tests survive a future reimplementation: - projection edge cases, including the segment-boundary guard that keeps root_path="/api" from slicing "/apifoo/models" into a string the router would never match - agreement with the router under nested mounts - the two bypasses these predicates exist to prevent: a protected route mounted under the "/health" public prefix must still 401, and a POST mounted under "/api/webhooks" must still require a CSRF token Both are verified to fail when the projection is reverted to `request.url.path` (9/13 red) and when a plausible vendored copy omits the boundary guard (the 2 boundary cases red). Declare starlette as a bounded direct dependency so a bump — which is security-relevant here — shows up in review rather than arriving silently through FastAPI. * ci: pin uv to the version production ships ExtensionManager is not a consumer of uv the build tool -- it is a program whose whole job is driving `uv` as a subprocess, depending on its CLI behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into `[dependency-groups] extensions`) and on the `uv.lock` serialization format. uv is closer to a runtime dependency with a contract than to incidental tooling. backend/Dockerfile pins that binary to 0.11.1, but all eight astral-sh/setup-uv steps installed whatever was latest at run time, so CI exercised the manager against a uv that is not the uv production runs. The sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI stays green because the same uv reads back what it wrote, and the pinned uv in the production image cannot read the committed lock. `uv lock --check` is version-sensitive for the same reason -- it verifies the lock is what *this* uv would produce, and two versions can emit equivalent but non-identical output. Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so the steps share input and caching behavior. Pinning alone drifts apart again on the next bump, so add a constraint test in the style of test_compose_default_bind_host.py: the Dockerfile's UV_IMAGE tag is the single source of truth, and both compose defaults plus every setup-uv step must match it. Verified to fail when a pin drifts, when a step omits `version`, and -- the real scenario -- when the Dockerfile is bumped alone, which lights up the workflows and both compose files at once. * fix(gateway): state the extension route auth limit and abort a failed dev sync Two scoped review follow-ups. README: contributed routers cannot enter the host's reserved public prefixes, which makes every extension endpoint session-authenticated -- there is no way to expose an unauthenticated route. The rejection rule was documented but its consequence was not, so inbound provider webhooks and public status endpoints read as merely undocumented rather than out of scope for this release. docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e` already stopped the script there -- uvicorn was not being started against a stale environment -- but it exited on a bare uv exit code with no indication of what to do. Abort explicitly with the cause and the fix. Tests slice the sync block out of the real script and run it against a stub uv, so they exercise the shipped code rather than a copy of it (/app/backend only exists inside the container). They cover the success path, the retry that recovers, the abort, and the guidance. Verified against the pre-fix script: only the guidance case goes red, confirming the abort itself was already correct. * fix(extensions): point Git SSH shorthand at the HTTPS correction Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git` reached the scheme rules looking like a bare path and was rejected with "local path references are not deployable; pass a local directory so DeerFlow can snapshot it". The operator asked for a remote source, so that guidance points at the wrong fix. Detect the shorthand ahead of the scheme rules and report the public-HTTPS correction instead. The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose `host` reads as a URL scheme and produced the generic HTTPS message. Both spellings now share one message, as does the PEP 508 named form. * docs: keep the root extension summary within its new budget #4799 split the depth out of the module guides and added a size gate; the root file's job is now orientation, and this branch had pushed it 192 bytes past the soft limit. The manager transaction, source rules, and lock discipline are already stated in full in the extensions guide, so the root keeps the one-line orientation and points there instead of restating them. --------- Co-authored-by: Codex <codex@openai.com>	2026-08-13 23:55:30 +08:00
ajayr	6cbf20fd39	feat(memory): add Honcho backend (user-model memory provider) (#4730) ... * feat(memory): honcho backend config parsing Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(memory): honcho v3 http client Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(memory): honcho memory manager (workspace-per-user, fail-closed identity, async offload) - HonchoMemoryManager implements the MemoryManager contract (add/get_context/ search/get_memory/shutdown_flush + aadd/aget_context/asearch offloaded via asyncio.to_thread), signatures verified against manager.py's tier-1/tier-2/ async abstracts. - Workspace resolution: workspace_overrides[user_id] else workspace_prefix + sanitize_id(user_id); missing/empty user_id fails closed (no-op write, empty read) rather than falling back to a shared workspace. User peer: user_peer_overrides[user_id] else sanitize_id(user_id). - get_context self-truncates to max_injection_chars and raises MemoryManagerError only under failure_policy.read=fail_closed; default is log-and-return "". - Restore backends/honcho/__init__.py to the noop direct-import convention (MANAGER_CLASS = HonchoMemoryManager) now that honcho_manager.py exists, replacing Task 10's temporary lazy __getattr__ scaffold. - Fix Task 10 deferred docstring minor: sanitize_id docstring now states the grammar allows up to 100 chars while this helper caps at 64. - 19 new tests appended to test_honcho_memory_backend.py (write/read/async/ lifecycle/factory-discovery); 27/27 pass. Verified end-to-end that manager.py's drop-in backend scanner resolves "honcho" with no core edits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(memory): collision-resistant identity derivation, exception containment, passive-writes flag Task review findings (2 Critical + 1 Important), all fixed in the same worktree: - CRITICAL (cross-user bleed): sanitize_id is lossy -- "user.name@example.com" and "user-name@example.com" both sanitized to the same string, merging two users' memory into one workspace/peer. Add _stable_id() (sanitize_id output + 8-hex-char SHA-256 suffix of the raw id) and use it on the default (non-override) path in _workspace/_user_peer; workspace_overrides / user_peer_overrides still match on the raw key, unchanged. The hash suffix also guarantees a non-empty result for a raw id that sanitizes to "" (e.g. "!!!"), so _user_peer can no longer return "". Documented in the manager's isolation docstring. - CRITICAL (exception containment): client.py's _post() called response.json() outside the try block, so a 200 with a non-JSON body raised a bare JSONDecodeError that would escape add() with no upstream handler. Wrap the parse and raise HonchoRequestError (mirrors Mem0Client._request). Broadened the manager's four boundary excepts from `except HonchoRequestError` to `except Exception` (mirrors openviking_manager.py's broad-guard precedent), with `except MemoryManagerError: raise` first so a contract error is never swallowed or double-wrapped. - IMPORTANT: added requires_passive_writes_in_tool_mode: ClassVar[bool] = True -- Honcho's only write path is passive add() (no fact CRUD hooks), so tool mode must keep MemoryMiddleware writes flowing to the deriver. Mirrors mem0_manager.py's identical flag/rationale. Minors addressed: get_memory(user_id=None) empty-shape-with-no-calls test; empty-string user_id tests for add()/get_context(); dedicated collision test proving two colliding raw ids resolve to different workspaces/peers. 10 new tests (37/37 total pass); RED verified by stashing only the implementation files (tests import the not-yet-existing _stable_id, so the whole module fails to collect) before restoring the fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(memory): blocking-io anchor for honcho backend; docs + config example - Adds test_honcho_memory_backend.py in tests/blocking_io/ with fake-client blocking IO - Mirrors openviking anchor structure and conftest conventions - Updates backends/README.md with honcho row and config keys section - Updates config.example.yaml with honcho commented block - Updates backend/AGENTS.md with honcho memory backend bullet - Documents workspace resolution (prefix + collision-resistant sanitized id) - Documents tool mode passive write retention via MemoryMiddleware - Documents async entrypoint offloading via asyncio.to_thread - Documents fail_closed vs fail_open recall failure policy Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(memory): wire close() to shutdown hook; correct honcho README defaults and tool-mode note - HonchoMemoryManager.close() releases the HTTP client, mirroring mem0_manager.py's pattern and the base MemoryManager.close() shutdown hook. - README: fix workspace_prefix (deerflow-u-), message_char_limit (8000), max_injection_chars (6000), and base_url (default http://localhost:8000, not required) against backends/honcho/config.py; add missing timeout_seconds/connect_timeout_seconds rows; replace the "middleware mode only" claim with wording matching reality (tool mode supported, search implemented, passive writes retained via MemoryMiddleware). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(memory): honor failure_policy.read on all honcho recall paths; review nits Addresses PR #4730 review feedback: - search() and get_memory() now route through a _read_or_fallback policy gate (mem0's pattern), so failure_policy.read: fail_closed raises MemoryManagerError on every recall path as documented; get_context() uses the same helper, preventing future drift. - Session ids use the collision-resistant _stable_id derivation; bare sanitize_id would merge threads like "t.1"/"t-1" into one session. - HonchoClient accepts a transport kwarg (Mem0Client precedent) so tests inject httpx.MockTransport through the constructor. - Config: empty/null workspace/peer override values fail fast at parse time instead of silently falling through to the default derivation. - _UTC_NOW_FIELDS 1-tuple replaced by a plain _UTC_NOW_FORMAT constant. - README: user_peer_overrides row described the wrong target (it overrides the user's own peer, not assistant_peer); document the non-empty constraint on override values. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(memory): qualify honcho isolation claim for shared workspace_overrides The module docstring claimed users cannot see each other's memory by construction, unconditionally. That holds for the default one-workspace-per-user derivation, but a workspace_overrides entry mapping several users to one workspace shares that workspace's search index: search() uses Honcho's workspace-scoped /search (no peer filter), while get_context()/get_memory() stay peer-scoped via working_representation. State the asymmetry in the docstring, the README Workspace Resolution section, and the workspace_overrides table row. Docs-only; no behavior change (review follow-up on #4730). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-12 09:02:08 +08:00
Aari	e9387394bc	feat(mcp): add durable task runtime foundation (#4665) ... * feat(mcp): add durable task runtime foundation * fix(chart): sync embedded config version * fix(mcp): isolate task polls during shutdown * feat(mcp): track consecutive poll errors on mcp_tasks poll_attempt_count grows on every claim (successful polls included), so it cannot drive a failure backoff without misjudging normal long tasks. Add consecutive_poll_error_count: incremented when a claim is released after a poll error, reset to zero by any applied snapshot. The backoff/terminal policy that consumes it lands with the first concrete driver. * fix(mcp): harden durable task lifecycle * fix(mcp): preserve tracked task on dedup conflict --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-08-08 20:03:36 +08:00
Hao Zhe	6556d09d7f	refactor(memory): use official OpenViking adapter (#4707) ... * refactor(memory): use official OpenViking adapter * fix(memory): preserve OpenViking recall behavior * fix(memory): ignore ambient OpenViking headers	2026-08-07 11:22:00 +08:00
ajayr	d732b90dc3	feat(channels): add Buzz (Nostr) channel connector (#4649) ... * feat(channels): add Buzz (Nostr) channel connector Adds a Buzz (https://github.com/block/buzz) channel so DeerFlow can join a Nostr-relay workspace as a member: it answers @mentions in channels, replies to DMs, and streams answers by editing one message in place. * app/channels/buzz_nostr.py — pure NIP-01 helpers: canonical event ids, BIP-340 signing/verification, chat/edit/auth builders, relay frames. * app/channels/buzz.py — BuzzChannel: one NIP-42-authenticated websocket, channel discovery (kind 39000) with one subscription per channel, live membership tracking (44100/44101), per-channel replay watermarks, and replies posted once then edited in place (kind 40003). * app/channels/buzz_run_policy.py — same-thread serialization, mirroring the Feishu precedent. Inbound is gated in order: signature verification, self-drop, /connect bind-and-return, pubkey allowlist, then mention / DM / mention-free / thread-follow. Off by default; needs the new optional `buzz` extra (coincurve, lazily imported), which detect_uv_extras resolves from channels.buzz.enabled the same way it already handles channels.discord. Two relay behaviours drove the design and are worth knowing when reviewing: a global {"kinds":[9]} subscription receives nothing from buzz-relay and a multi-value "#h" filter receives nothing either, so one REQ per channel is required; and a single global `since` cursor skips quiet channels, so watermarks are per channel. Signed-off-by: Ajay R <ajayr@formbuddy.com> * fix(channels): only publish assistant messages from the IM stream `_accumulate_stream_text` decided what streamed `messages-tuple` payloads become displayable text by rejecting ONLY payloads whose `type` contained "tool", so it published everything else. DeerFlow writes hidden model context into the messages channel as ordinary messages -- memory recall and the rewritten user turn as hidden HumanMessages (DynamicContextMiddleware), the `<durable_context_data>` block as another (DurableContextMiddleware) -- and LangGraph fans those state writes out on the messages stream, so they reached every streaming IM channel as the assistant's reply. Proved live on a Buzz relay: the connector published a `<memory>` fact block and, in another run, a verbatim echo of the user's own inbound message. Affects Feishu, Telegram, WeCom and Buzz; worst on Buzz, where each update is an immutable public Nostr event that a corrective edit cannot unpublish. Invert the filter to an allowlist of assistant message types. Two new pure helpers keep it testable: - `_stream_payload_type` resolves the type from both shapes the function already handles: the `model_dump()` shape the gateway emits, and LangChain's `to_json()` constructor shape whose own `type` is the literal "constructor" and whose class name is the tail of the `id` path. - `_is_assistant_stream_type` matches "ai"/"assistant" by PREFIX, not substring -- ordinary words contain "ai" ("chain", "domain"), and a substring test would admit a foreign type name by accident. The bare-`str` branch is removed: an untyped payload cannot be attributed to the assistant, nothing in DeerFlow produces one (serialize_messages_tuple always emits `[message_dict, metadata]`), and a runtime that emitted raw text deltas would emit hidden context the same way. Per-message-id buffering and merging are unchanged. Tests pin both directions, including multi-chunk merging across one message id, so the allowlist cannot silently kill streaming, plus an end-to-end `_handle_streaming_chat` test asserting the live payload never reaches an outbound message. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Ajay R <ajayr@formbuddy.com> * chore(helm): bump config_version to 33 in chart values and README config.example.yaml moved to 33 for the buzz channel block; the chart's embedded config example and its README copy track it (config_version only drives the outdated-config warning, per scripts/check_config_version.sh). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ajay R <ajayr@formbuddy.com> --------- Signed-off-by: Ajay R <ajayr@formbuddy.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-08-05 08:29:22 +08:00
Nan Gao	1f792d0f4b	feat(extensions): add middleware plugin foundation (#4636) ... * feat(extensions): add middleware plugin foundation * fix(extensions): stop config resolution from masking extension loading `create_app()` resolved the configured plugin list inside the fail-open guard around `load_extensions()`. CI has no `config.yaml` (gitignored and never generated by the workflow), so `get_app_config()` raised `FileNotFoundError` there and was swallowed as an extension failure -- `load_extensions()` never ran at all, and the four `create_app()` tests in `test_extension_app_loading.py` passed locally but failed on every runner. Resolve the plugin list before the guard. Only an absent `config.yaml` is tolerated, mirroring `_resolve_trace_enabled_for_app_construction()`: `create_app()` runs at import time, and lifespan still performs strict config loading before serving. A `config.yaml` that exists but fails to parse or validate now propagates instead of being reported as an extension failure -- reporting it as the latter silently dropped a `required: true` extension rather than failing the boot. Make the tests config-independent with an autouse `stub_app_config` fixture, following the existing pattern in `test_gateway_lifespan_shutdown.py`, and cover both new branches of the config-resolution boundary. * fix(extensions): bind the run's extension snapshot through subagent delegation The lead-agent path resolves one immutable loaded-extension snapshot per run and binds it through task-store allocation and graph construction, but the subagent path re-read the process-wide singleton at execution time. In production both are the same object, yet a `set_loaded_extensions()` between the lead run's start and a subagent's execution (test teardown, a future hot-reload path) would let one run mix two extension generations — exactly what the documented invariant exists to prevent. The graph-build binding is a ContextVar scoped to synchronous construction, so it has already exited by the time a tool delegates; the snapshot has to travel through runtime context instead. The run worker publishes it under the host-internal `EXTENSION_SNAPSHOT_CONTEXT_KEY` (written after the caller merge, popped when the run has none, so a caller-supplied value is never authoritative), `task_tool` reads it back through the type-checking `resolve_run_extensions()`, and `SubagentExecutor` binds it at construction. Callers outside the Gateway run path — embedded `DeerFlowClient`, standalone LangGraph Server — install no snapshot and keep the existing `get_loaded_extensions()` fallback. * refactor(extensions): defer the ordering table by call, not by a lying tuple `CORE_ORDERING_CONSTRAINTS` was a `tuple` subclass that overrode only `__iter__` and resolved into a class-level `_resolved` side channel. A tuple cannot populate its own storage after construction, so the instance stayed the empty tuple it was built as: `len()` was 0, `bool()` was False, `in` was always False, indexing raised, slicing and `reversed()` came back empty, and it compared unequal to the plain tuples tests substitute for it — all while iteration yielded the real constraints. Only `assert_ordering` consumed it, and only by iterating, so the split went unnoticed. The sibling `_AnchorTable(dict)` uses the same idea soundly because dict is mutable: `self.update()` fills the real storage, making every inherited operation correct. That trick does not survive the port to an immutable type. Replace it with `core_ordering_constraints()`, matching how `stack.py` defers the same kind of table via `_anchors()`. The deferral is kept — it is about dependency direction, not just cycles: `extensions/` is the layer the middleware layer calls into, so a module-scope `agents.middlewares` import here points the dependency backwards and closes a cycle as soon as any middleware imports something under `extensions/` at module level. Resolution stays at `assert_ordering` time, which already runs inside the middleware builder. Tests pin both halves: the returned value is a plain tuple whose len/bool/ membership/indexing/reversal/equality agree with iteration, and a subprocess probe asserts importing `extensions.ordering` does not load the middleware layer while calling the function does.	2026-08-04 22:33:26 +08:00
Vanzeren	c8cf1bf2fb	feat(checkpoint): checkpoint history cache (#4638) ... * feat(checkpoint-cache): delta-mode checkpoint history cache with recursive compose Read-only, invalidation-free cache for LangGraph delta-channel history ({writes, seed}) at the get_delta_channel_history choke point: - database.checkpoint_cache config (memory|redis; max_entries 0=disabled; redis bounded by TTL, Gateway/async only) - memory LRU backend (copy-on-read, zero-serde hit path) and redis backend (lazy import, degrades to all-miss on outage) - CachedHistorySaver: recursive composition from the nearest warm ancestor (depth budget 8), caching each level; depth-0 cold chains delegate one inner fast-path walk. Entries keyed by immutable (db, thread, ns, checkpoint_id, channel) — no invalidation, coherent across workers - provider wiring: wraps in delta mode only (async + sync), full mode untouched; sync path is memory-only - bench opt-in: DEERFLOW_CHECKPOINT_BENCH_HISTORY_CACHE=1 sqlite bench (500 updates, payload 2KB): write phase 2.28x at f=250, 1.32x at f=10; one delegated walk per thread cold start. * chore(config): bump config_version to 32 for database.checkpoint_cache The checkpoint history cache feature added the database.checkpoint_cache section to config.example.yaml; bump the schema version so existing deployments get the outdated-config warning and can run make config-upgrade. * chore(helm): bump config_version to 32 in chart values and README * fix(checkpoint-cache): purge thread history entries on delete paths Addresses review on #4638: delete_thread/prune removed source-of-truth checkpoints but left the thread's materialized history payloads in the cache (memory: until LRU eviction; redis: until TTL, default 1 day) — a data-lifecycle gap for tenant offboarding / GDPR-style erasure. - Cache contract gains thread-scoped adelete_thread/delete_thread (lifecycle purge, not invalidation; entries remain immutable) - Memory backend: stem scan over the LRU map; redis: SCAN MATCH + UNLINK, outage degrades to TTL-bounded retention without raising - CachedHistorySaver purges on delete_thread/adelete_thread and prune/aprune (prune rewrites chains, so pre-prune histories must go); delete_for_runs stays delegation-only (run->thread mapping unavailable, no in-tree callers), documented in code - ttl_seconds description documents the residual-retention window - Tests: thread-scoped purge on both backends, saver-level delete/prune purge, prefix-safety (t1 vs t10), redis outage degradation, and the pinned no-purge behavior of delete_for_runs * fix(checkpoint-cache): stable db identity, prefix-aware sync singleton, explicit zero TTL Addresses Copilot review on #4638: - checkpoint_cache_db_hash now hashes the credential-free postgres identity (host:port/database + schema): credential rotation no longer changes the cache namespace (cold cache + orphaned keys until TTL). Unparseable URLs fall back to the raw string. - The sync-path memory cache singleton is also keyed by its key_prefix: a namespace change (db identity change or operator override) recreates the cache instead of leaving stale-prefix entries unreachable and unpurgeable. - ttl_seconds=0 is now an explicit, documented opt-out of redis expiry (SET without EX; redis maxmemory policy only) instead of a silent 'ttl_seconds or None' coercion. Tests: credential-rotation hash stability, unparseable-URL fallback, prefix-change singleton recreation, same-prefix singleton reuse, and zero-TTL wire behavior (ex=None). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-08-02 22:25:02 +08:00
hataa	540940bac1	feat(authz): enforce model authorization at Gateway routes and runtime (#4063 Phase 3) (#4540) ... * feat(authz): enforce model authorization at Gateway routes and runtime (#4063 Phase 3) Phase 3 / Models — the first of three resource-type PRs (Models, Skills, Sandbox). The RBAC provider already maps "model" → config key "models" (rbac.py _RESOURCE_POLICY_KEYS), so no schema change is needed. Gateway route layer (mirrors Phase 2A): - resolve_model_authorization() in authz.py returns (provider, principal), reusing _get_cached_route_provider and build_principal_from_context, including the INTERNAL_SYSTEM_ROLE → None pop for internal callers. - list_models filters via provider.filter_resources(principal, "model", names). - get_model checks provider.authorize("model", "use"). Deny → 403 (not 404, since the model exists but the role lacks permission). Runtime resolution layer (mirrors Phase 1B): - _authorize_model_name() in agent.py runs after _resolve_model_name. On deny, falls back to the first allowed model (RFC §9: graceful, not crash). All models denied + fail_closed → ValueError (matches existing contract). authorization.enabled: false is a complete no-op on both layers. Anonymous requests (user=None) bypass filtering. 18 new tests + 314 existing tests pass. * fix(authz): enforce model:use on the embedded DeerFlowClient path (Phase 3 follow-up) Round 4 review (willem-bd): _authorize_model_name only covered the Gateway runtime path (_make_lead_agent). The parallel lead-agent construction path DeerFlowClient._ensure_agent (client.py) filtered tools but not the model, so a library/embedded consumer with role-scoped model policies could run a model the role is denied model:use for. - Insert _authorize_model_name in _ensure_agent, mirroring _make_lead_agent. - Resolve None default to the first configured model before the gate so the implicit default (create_chat_model(name=None)) is also authorized. - Update test_authorization_filters_framework_tools_and_reuses_provider: the stub provider now returns an allow decision for model:use (checked during assembly) and patches resolve_authorization_provider in the agent namespace. - Add 3 DeerFlowClient._ensure_agent path tests (real-path fallback, None-default resolution, disabled no-op); 24 tests total. * docs(authz): document get_model provider-unavailable fail-open path + test zhfeng review (round 5): get_model's docstring only mentioned the deny→403 path, not the provider-resolution-error + fail-open path (which allows the request, mirroring list_models's documented fail-open semantics). The behavior itself is correct and symmetric with list_models, but it was undocumented and the _AuthorizationUnavailable path had no test coverage. - Extend get_model docstring to state the provider-error fail-closed/fail-open outcome, matching list_models's wording. - Add test_get_model_provider_unavailable_fail_closed_vs_open exercising the _AuthorizationUnavailable path (provider cannot be resolved at all), pinning fail-closed→403 / fail-open→200. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-08-01 22:34:11 +08:00
Amorend	85c3909c2e	feat: show real-time context window usage (#3125) (#3183) ... * feat: show real-time context window usage in chat UI (#3125) Adds a `context_usage` block to `GET /api/threads/{id}/token-usage` (token count from the live checkpoint, the thread model's `context_window`, and a percentage), introduces a new `ModelConfig.context_window` distinct from the per-call `max_tokens` output cap, and surfaces the percentage in the chat header — inside `TokenUsageIndicator` when token-usage tracking is on, or as a standalone badge when it's off so context capacity stays visible independent of cost tracking. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: per-category breakdown for context window usage Replace the single-number context_usage payload with a Claude-Code-style breakdown — messages, system prompt, skills, system/MCP tools (active + deferred), custom agents, memory injection, autocompact buffer, and free space — and surface it in the chat UI with a segmented progress bar and per-row table. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(config): document context_window across model examples Add `context_window` to every example model in config.example.yaml so the new chat-UI "% context used" indicator works out of the box for whichever example a user adopts. Each value is the published default at the time of writing; users are pointed at the official model spec to verify. Bumps config_version to 11 so `make config-upgrade` flags outdated user configs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * style: ruff format (line-length 240) No behavior change — collapses two multi-line expressions that fit on one line under the project's 240-char limit. Picked up by `make format`. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * review: address Copilot bot comments on #3183 - token-usage-indicator: switch `{contextPercentage && (...)}` to an explicit `!= null` check. (The string `"0"` is actually truthy in JS so the original code wasn't buggy, but the explicit check is clearer.) - context-usage-breakdown: drop the `useMemo` around segments/totals — the computation is O(n) over a handful of rows and the previous memo deps omitted `t.contextUsage.categories`, so the bar's tooltips/aria-labels could stay in the old language after a locale switch. - context_usage._split_tools: snapshot MCP names from `get_cached_mcp_tools()` directly instead of re-reading `extensions_config.json` after `get_available_tools()` already loaded it. Removes redundant file I/O on every `/token-usage` poll. (`get_available_tools()` still emits its own INFO logs — silencing those is out of scope here.) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * style(frontend): prettier --write context-usage-breakdown CI's `pnpm format` (prettier --check) caught two lines previously formatted by hand. Collapses one comma to fit on one line; no behavior change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(gateway): correct context-usage breakdown + add exact token counting The context-usage indicator shipped two bugs that silently zeroed whole breakdown rows (both caught by try/except, so the feature looked alive but produced wrong numbers): 1. _count_system_prompt passed app_config= to get_deferred_tools_prompt_section, which only accepts deferred_names -> TypeError swallowed -> system_prompt row always 0, and used_tokens/percentage undercounted by the full prompt. Also subtracted the deferred section twice (the rendered prompt already excluded it). Fix: derive deferred names deterministically and pass them to apply_prompt_template; drop the redundant subtraction. 2. _split_tools imported a non-existent get_deferred_registry -> ImportError swallowed -> all four tool-category rows always 0. Fix: classify via the public is_mcp_tool predicate + tool_search.enabled (mirrors build_deferred_tool_setup); the MCP tag is set by get_available_tools. Added token_usage.counting (approximate|exact). 'exact' routes text/schema/ message counting through the model tokenizer (tiktoken cl100k_base) via the existing memory-module machinery (lazy load + cache + cooldown + CJK-aware fallback), so CJK-heavy threads stop being undercounted by chars//4. Regression + e2e tests added; 6621 backend tests pass. * fix(gateway): harden context usage accounting * fix(gateway): count promoted MCP tools as active in context usage Promoted tools (deferred MCP tools the thread has fetched via tool_search) have their full schema bound on every subsequent turn by DeferredToolFilterMiddleware, so they consume context like any active tool. The breakdown previously left them in the reserved *_deferred rows, under- counting the thread's used_tokens. Classification now treats a tool as deferred only when tool_search is enabled, it is MCP-sourced, AND it has not been promoted. The promoted set is read from the checkpoint's channel_values and scoped by catalog hash — matching the runtime middleware, so a stale promotion from MCP-config drift cannot inflate the active count. The static system prompt still lists all deferred tool names (promotions only affect schema binding, not the prompt), so _count_system_prompt's deferred rendering is intentionally left unchanged. 8 new tests cover classification, catalog-hash scoping (match / drift / compute-failure / malformed), and checkpoint extraction. * fix(context): address review feedback * fix(context): count structured message payloads * fix(context): harden usage accounting * fix(config): bump schema for context usage fields * refactor: narrow context usage to core indicator --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-31 21:57:22 +08:00
Ryker_Feng	063d62c3c3	feat(persistence): support custom postgres schema (#3442) ... * feat(persistence): support custom postgres schema * fix(persistence): address CI lint/test failures and review feedback - Map missing psycopg import to actionable POSTGRES_INSTALL guidance in sync/async schema-creation helpers - Accept SQLAlchemy compound DSN schemes (postgresql+asyncpg) when injecting search_path, normalizing to a libpq-consumable DSN - Guard keyword-DSN tests with importorskip so they skip without psycopg - Set database=None in sync checkpointer none-fix test to avoid MagicMock backend resolution - Apply ruff import sort and format * fix(persistence): address pg-schema review feedback - Restrict postgres_schema regex to lowercase-only so the quoted CREATE SCHEMA matches the unquoted search_path (PG case-folds it), fixing the mixed-case bug where tables silently fell back to public. - Replace shlex.join/split with libpq-correct backslash escaping for the options parameter so values containing spaces survive intact. - Add normalize_libpq_dsn() and route the async checkpointer pool through dsn_with_search_path() so a +asyncpg suffix is stripped and existing DSN options (e.g. statement_timeout) are merged instead of overridden. - Extract shared ensure_postgres_schema()/ensure_postgres_schema_async() helpers (mapping missing psycopg to the install hint) used by all four provider sites. - Tests: reject mixed-case schemas, preserve space-containing libpq option, cover normalize_libpq_dsn, and assert pool search_path via DSN. * fix(persistence): align pg-schema test with merged store API The main merge moved the sync Store factory to the single-path _resolve_store_config/_sync_store_cm design, dropping the PR's _sync_store_from_database helper. The integration test still imported the removed symbol, breaking test collection (backend-unit-tests). Resolve the store config from a DatabaseConfig and drive it through _sync_store_cm instead. * fix(persistence): address pg-schema review feedback - reject trailing/leading whitespace in postgres_schema via re.fullmatch (a $-anchored re.match let "deerflow\n" through, silently landing tables in public) - re-escape all whitespace (TAB/CR/LF) when re-joining libpq options so a caller's pre-existing options value round-trips losslessly - re-validate the identifier inside create_schema_sql as defense-in-depth at the SQL-emitting boundary - accept the postgres:// short scheme in the alembic search_path injection - close the sync psycopg connection explicitly (psycopg3 __exit__ does not close()), mirroring the async path - drop the partial checkpointer/store reset on a database config change; database is restart-required and the ORM engine is not rebuilt, so a partial reset would half-migrate the deployment * docs(config): complete the postgres_schema migration checklist Address PR review (P1): the documented `public`->schema migration only moved runs, run_events, threads_meta, feedback, and users. That strands every other DeerFlow-owned table -- the four channel_* tables, both scheduled_* tables, agents, and (critically) alembic_version -- in `public`. On restart bootstrap treats the partially-populated target schema as unversioned, re-baselines it, and replays migrations while the real rows stay invisible in `public`. List the full owned set explicitly, call out alembic_version as required, and keep the "discover the rest" query for version-drift safety. * refactor(checkpointer): drop test-only _sync_checkpointer_from_database Address PR review: the helper was only reached by the env-gated integration test and re-implemented the DatabaseConfig->CheckpointerConfig backend resolution that _resolve_checkpointer_config already owns, so a future backend added there would silently miss this path. Mirror the store side of the same test, which reuses the production path directly: _resolve_checkpointer_config(...) + _sync_checkpointer_cm(...).	2026-07-30 13:51:17 +08:00
Ryker_Feng	594dbe1205	fix(console): disable cost reporting when model pricing mixes currencies (#4564) ... The console cost estimator summed per-run spend across every priced model without checking that they shared a currency, so a deployment that priced one model in CNY and another in USD produced a meaningless aggregate under a single currency label. Track the first priced currency and, on a mismatch, log a warning and drop all pricing so cost/currency fields report null instead of an invalid sum. Currency codes are now trimmed and case-normalized so equivalent spellings don't false-trip the guard.	2026-07-29 22:24:04 +08:00
Daoyuan Li	9bb8225079	fix(memory): harden OpenViking retries and watermarks (#4552)	2026-07-29 07:24:55 +08:00
Vanzeren	352f247a81	feat(memory): add mem0 HTTP memory backend (#4528) ... * feat(memory): add mem0 HTTP memory backend * fix(memory): address mem0 review feedback --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-29 07:11:20 +08:00
阿泽	9c7cd4cad3	feat(sandbox): add thread data mount override for upload sync (#4536)	2026-07-28 23:41:14 +08:00
RongfuShuiping	2aaf74b0f8	feat(memory): add OpenViking HTTP backend (#4509) ... * feat(memory): add OpenViking HTTP backend * fix(memory): harden OpenViking lifecycle --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-28 23:36:25 +08:00
Aari	a5059b8284	fix(subagents): isolate callbacks and activate skills lazily (#4497)	2026-07-28 23:29:14 +08:00
Vanzeren	c48de5e70b	feat(checkpoint): make delta snapshot_frequency configurable (#4516) ... * feat(checkpoint): make delta snapshot_frequency configurable * fix(config): carry legacy checkpoint_delta_snapshot_frequency with warning Addresses review on #4516: the rename from the flat database.checkpoint_delta_snapshot_frequency key to nested database.checkpoint_delta.snapshot_frequency silently dropped the old value (pydantic extra="ignore"). Add a before-validator that maps the legacy key onto the nested one with a deprecation warning (nested key wins when both are set), plus a CHANGELOG breaking-change note covering the rename and the 1000 -> 10 default change. * fix(checkpoint): validate frozen snapshot frequency	2026-07-28 23:21:23 +08:00
阿泽	94003c1f47	feat(models): support cumulative vLLM stream usage (#4537) ... * feat(models): support cumulative vLLM stream usage * fix(models): preserve active cumulative usage streams	2026-07-28 19:56:40 +08:00
Ryker_Feng	0a9ce5d7e3	feat(suggestions): configure follow-up suggestion count (#4533)	2026-07-28 19:35:21 +08:00
ajayr	6456c35675	fix(browserless): accept the timeout config key and harden coercion (#4519) ... `browserless` reads `cfg["timeout_s"]`, while its sibling web providers `crawl4ai` and `jina_ai` read `cfg["timeout"]`. Tool configs allow extra fields, so the unrecognised spelling is dropped without a diagnostic: someone adapting one provider's config snippet for another silently gets the 30s default instead of the timeout they set. (Observed in the other direction, on a deployment whose crawl4ai entry carried `timeout_s`.) Accept both keys, preferring the documented `timeout_s` when both are present. While adding coverage, two pre-existing bugs in the same three lines surfaced, both already guarded in crawl4ai/jina_ai but not here: - `timeout_s: "30s"` (or any non-numeric string) raised ValueError out of `float(raw)` during tool construction rather than falling back. - `timeout_s: off` -- YAML parses that as `False`, and `float(False)` is `0.0`, so every request timed out immediately against a healthy server. `_coerce_timeout` now mirrors the sibling providers: booleans and unparsable strings fall back to the default, with a warning for the string case. Tests: five cases in tests/test_browserless_client.py covering both keys, the precedence order, and both coercion bugs. Verified red before the fix (3 of 5 fail) and green after. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-07-28 07:56:04 +08:00
ajayr	28553040fe	docs(config): Crawl4AI >= 0.9 requires a bearer token (#4518) ... The commented Crawl4AI web_fetch example still describes the pre-0.9 server: it says JWT auth is off by default, marks `token:` as only needed "if the server has JWT auth enabled", and pins 0.8.6 in the docker run line. Crawl4AI 0.9.0 made the Docker API server secure-by-default. Auth is on for every request except GET /health, and a server started without CRAWL4AI_API_TOKEN binds 127.0.0.1 only. Following the current example against a current image therefore yields either HTTP 401 on every fetch, or a server a containerised DeerFlow cannot reach -- with no hint that auth is the cause. 0.8.6 is also worth moving off: 0.8.7 fixed two pre-auth RCEs (CVSS 9.8), and 0.8.8/0.8.9 closed SSRF gaps in the same server. No code change is needed -- Crawl4AiClient already sends `Authorization: Bearer <token>` whenever `token` is set, so this is purely the example catching up with the upstream server. Comment-only, so config_version is unchanged. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-07-28 07:44:59 +08:00
qin-chenghan	795af20a6b	feat(memory): built-in FTS5/BM25 retrieval adapter (#4360) ... * feat(memory): integrate FTS5 retrieval adapter * deps: add jieba as default dependency for Chinese tokenization Without jieba, FTS5 unicode61 tokenizer treats entire Chinese sentences as single tokens, making single-character or sub-phrase searches impossible (e.g. '吃' or '油泼面' returns 0 hits against '用户喜欢吃油泼面'). jieba segments Chinese text into meaningful tokens before indexing. * fix(memory): avoid treating hyphens as FTS5 operators * feat(memory): make Chinese tokenization optional * fix(memory): warm every requested retrieval scope * fix(memory): close retrieval resources on shutdown * fix(memory): close backend when shutdown flush fails * fix(memory): recreate corrupt retrieval index * fix(memory): tolerate partial retrieval rebuilds * fix(memory): warm retrieval index in background * fix(memory): preserve shutdown flush budget * fix(memory): stop retrying partial lazy rebuilds * fix(memory): close retrieval through storage * refactor(memory): simplify retrieval scope limit * docs(memory): clarify retrieval shutdown lifecycle --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-27 23:17:18 +08:00
Zhipeng Zheng	838037188e	feat(channels): share inbound webhook dedupe across pods via Postgres (#4210) ... * feat(channels): share inbound webhook dedupe across pods via Postgres (#4120) * ci: run cross-pod inbound dedupe integration tests in CI Expose the job Postgres service via DEDUPE_TEST_POSTGRES_URL so the integration tests (issue #4120) actually execute instead of silently skipping. Normalize the URL for asyncpg (postgresql:// -> +asyncpg, drop libpq-only sslmode) and await the now-async _is_duplicate_inbound in test_github_dispatcher.	2026-07-27 23:07:40 +08:00
Zybnev Sergey	a9a5fc9ced	fix(telegram): render final replies as Rich Messages (#4387) ... * fix: render Telegram replies as rich messages * Исправление fallback Rich Messages в Telegram	2026-07-27 22:51:02 +08:00
Diego Câmara	ac18f518c8	feat(sandbox): add Tenki cloud sandbox provider (#4382) ... * feat(sandbox): add Tenki cloud sandbox provider Adds deerflow.community.tenki, a SandboxProvider backed by Tenki cloud microVMs, alongside the existing e2b_sandbox / boxlite / aio_sandbox backends. Selected via `sandbox.use: deerflow.community.tenki:TenkiSandboxProvider` (resolved by class path, so the change is purely additive). The full Sandbox contract is implemented — execute_command plus read/write/update/download_file and list_dir/glob/grep — with file ops run as busybox-portable shell commands (cat / find / grep / chunked base64), reusing deerflow.sandbox.search, mirroring e2b_sandbox and boxlite. Tenki's SDK is synchronous, so unlike boxlite there is no event-loop bridge. Tenki sandboxes run as an unprivileged user with /mnt root-owned, so the /mnt/user-data virtual prefix is remapped under the writable home dir (like e2b_sandbox); the provider also best-effort sudo-symlinks /mnt/user-data to that home dir so agent shell commands using the literal path still work. Sandboxes are pooled per (user, thread) with warm reclaim, a replica cap, and an idle reaper via the shared WarmPoolLifecycleMixin. Transient transport blips get one bounded retry; terminal session errors evict and recreate. Only the stable Tenki surface is used (create/terminate + exec/shell/fs) — no volumes, snapshots, or template builds — so any stock base image works. The tenki-sandbox SDK is an optional extra (deerflow-harness[tenki]) and is imported lazily, so a default install and every other provider are unaffected. Tested: unit suite runs in CI without tenki-sandbox installed; a live integration test and full-surface e2e were verified against real Tenki sandboxes. * fix(sandbox): remove unsafe auto-retry from Tenki exec Pre-merge review caught that the transient-transport retry sat at the universal _exec layer, so it retried every operation — execute_command and base64 file-write chunks included. gRPC has no exactly-once guarantee: a "socket closed" ack-drop after the server already ran the op means the retry runs it twice, double-firing command side effects and duplicating a write chunk mid-file (silent binary corruption on multi-chunk writes). exec is not idempotent, so it must not be auto-retried. Reverts to the boxlite/e2b behavior: a transient error surfaces to the caller (returned as text by execute_command, raised by the file ops); a terminal session error still evicts the sandbox so the next acquire rebuilds it. Verified live end-to-end across 31 edge cases (empty/binary/unicode/chunk-boundary files, shell-metachar content, error paths, list/glob/grep, warm-pool reclaim, concurrency). * fix(sandbox): address Tenki provider review feedback - Use Tenki's native sandbox.fs API for all file transport (read_text, read_bytes, write_stream, mkdir) instead of cat/chunked-base64 over shell. Uploads stream in 1 MiB frames; append is read-modify-write because the write stream has no append mode (same approach as community/e2b_sandbox). - download_file streams via fs.read_stream and enforces the 100 MB cap on bytes actually received, closing the TOCTOU window between the old wc -c size probe and the read. - list_dir/glob/grep report paths back under /mnt/user-data instead of the sandbox-internal home dir, so results feed straight into the file APIs. - Create with wait=False and await wait_ready() here: create(wait=True) raises with the session handle still inside the SDK, leaking a running microVM this provider could never terminate. - Configure the sandbox lifetime (max_duration, default 4h) and expose sticky; without it Tenki reaps a reused thread's sandbox after ~30 min. - close() terminates before marking the adapter closed and re-raises real failures, so a failed termination stays retryable instead of silently leaking a billed microVM; an already-gone session still counts as closed. - Bump the optional extra to tenki-sandbox>=0.4.0 and commit backend/uv.lock. * fix(sandbox): scope tenki grep() glob filter to its directory prefix Mirrors #4168, which fixed the same defect in the E2B provider. The tenki adapter reduced a directory-scoped pattern like "src/*.js" to its basename before filtering, so the search silently broadened to every matching-extension file in the tree. Post-filter grep's hits through path_matches() against the path relative to the search root, the same way glob() already does, so both agree on what a directory-scoped pattern means. * fix(sandbox): address Tenki provider review — eviction, id width, write lock, grep -H Four fixes from the upstream review: download_file no longer swallows terminal transport errors. The broad `except OSError: raise` re-raised ConnectionError/BrokenPipeError/EOFError (all OSError subclasses that _is_terminal_failure treats as terminal) before _note_failure ran, so a session that died mid-download was never evicted. Only our own EFBIG size-cap now passes through without eviction. Sandbox id widened from 32 to 64 bits (`[:8]` to `[:16]`), matching community/e2b_sandbox. The warm pool is keyed by this id with no full-seed fallback, so a collision could let one user reclaim another's parked sandbox on a multi-tenant gateway. _fs_op now holds the lock across the op, not just the fs lookup, so concurrent calls on the same sandbox serialise over the SDK's shared connection. The eviction callback runs after the lock is released to avoid a lock-order deadlock with the provider. The append read-modify-write is serialised by a dedicated _write_lock so two concurrent appends can't clobber each other. grep passes -H so a search whose path resolves to a single file still prints the filename; without it the file:line:text unpack dropped every match. * fix(sandbox): address Tenki provider review round 2 - validate config `environment` at load time (_validate_extra_env) so a bad key fails fast instead of surfacing as an SDK error mid-command - document the deliberate lock decision in download_file: the instance lock is dropped before streaming so a 100 MB download can't block every other tool; terminal transport errors still evict via _note_failure - tighten the terminal-error comment to note ConnectionError/BrokenPipeError/ EOFError are also treated terminal via isinstance - document TenkiSandboxProvider in backend/AGENTS.md (provider detail, warm-pool destroy hook, community provider list) - add a commented Tenki block to config.example.yaml for parity with AIO/BoxLite - tests: config env validation, grep -F/case-sensitive flags, glob include_dirs, list_dir max_depth, bootstrap-failure warning branch * fix(sandbox): make Tenki bootstrap non-interactive and time-bounded The create-time bootstrap runs under the per-scope acquire lock, so a hang would stall acquire for that scope indefinitely: - use `sudo -n` so a password-requiring sudoers entry fails fast (swallowed by the existing `|| true`) instead of blocking on a tty password prompt - pass a timeout to the bootstrap `remote.exec` so any other stall drops to the existing warning path rather than wedging acquire Best-effort by design; the file APIs still work via the home remap on failure. * test(sandbox): pin Tenki bootstrap timeout to its actual value Assert bootstrap["timeout"] == _BOOTSTRAP_TIMEOUT instead of `is not None`, so a regression to timeout=0 (treated as no timeout by some SDKs) or an unrelated value is caught rather than passing a weaker non-None check. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-27 22:20:07 +08:00
hataa	6091ce7576	feat(authz): derive Gateway route permissions from AuthorizationProvider (#4439) ... * feat(authz): derive Gateway route permissions from AuthorizationProvider (Phase 2A, #4063) Phase 2A: replace legacy _ALL_PERMISSIONS with provider-derived route permissions. When authorization.enabled, each threads:*/runs:* permission is evaluated independently via provider.aauthorize(resource='route'). Disabled mode preserves legacy behavior. owner_check and require_admin_user remain unchanged. 10 new tests: disabled/enabled/RBAC policy/fail-closed/fail-open/ authenticate integration/middleware integration. * fix(authz): move AuthorizationConfig import to runtime for fallback in _get_route_authorization_config	2026-07-27 14:19:04 +08:00
Huixin615	090e80c1dd	fix(runtime): fail-stop runs when lease ownership cannot be confirmed (#4431) ... * fix(runtime): fail-stop runs after lease expiry * test(runtime): cover late successful lease renewal --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-27 07:25:34 +08:00
Huixin615	1cd5dea336	fix(streaming): signal replay history gaps (#4426) ... * fix(streaming): signal replay history gaps * fix(streaming): guard initial Redis replay window * fix(frontend): align inactive gap recovery --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-27 07:13:06 +08:00