325 Commits

Author	SHA1	Message	Date
creed	f4af75a7fd	feat(community): add bounded retries for Jina web fetch (#6143) ... * feat(community): add bounded retries for Jina web fetch Closes #6116 Keep retries opt-in and share a total deadline across requests and backoff. Preserve cancellation and the existing Error contract. Include offline DNS and local-wheel fixture corrections needed to run the complete backend suite in the isolated verifier. Signed-off-by: 97three <2212371308@qq.com> * docs(community): scope Jina retry guidance to its provider Signed-off-by: 97three <2212371308@qq.com> * fix(community): jitter bounded Jina retry backoff * docs(community): apply provider-scoped Jina guidance Signed-off-by: 97three <2212371308@qq.com> --------- Signed-off-by: 97three <2212371308@qq.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-02 21:35:56 +08:00
Totoro	b57fa6fb9e	feat(scheduler): push scheduled task results to IM channels via a notification outbox (#6135) ... * feat(persistence): add notification delivery outbox for scheduled task IM push New notification_deliveries table and repository (migration 0012) backing the outbox pattern for issue #4254: idempotent enqueue keyed on task_run_id plus event plus provider plus target, atomic claim of due rows, and exponential-backoff retry state moving pending to sending to sent or failed. Execution status stays in scheduled_task_runs; delivery status lives here. * feat(scheduler): enqueue and deliver scheduled-task notifications over IM channels Closes the loop for issue #4254. On run completion the scheduler writes outbox rows for the task owner's connected channel identities (success and failure events; interrupts stay silent), and a new NotificationDeliveryWorker polls due rows and pushes them through the channel's proactive send path. Channel gains send_notification; WeCom implements it via the WS client's send_message with errcode checking, the base class raises for providers without proactive push. Delivery is best-effort and never shadows execution status: enqueue and send failures are logged and recorded in the outbox, which retries with backoff. Channel liveness is re-checked at delivery time, and the worker stops before the channel service on shutdown. Feature activates only when channel connections are enabled; no new dependencies. Verified end to end against a live WeCom bot: scheduled run completion produced an outbox row that was delivered in under one second. * feat(scheduler): include scheduled run result summary in IM notifications The v1 notification only carried the run outcome skeleton (status, task id, run id). Completed runs now include the agent's final answer so the IM push is actually useful headlessly. - The delivery worker accepts an optional resolve_run_summary(run_id, user_id) resolver; for run_completed events it enriches the rendered text with a "Result:" line (bounded to 1000 chars). The lookup is best-effort: resolver failures or missing summaries degrade to the skeleton text and never block or fail the delivery. - The summary is read at delivery time from runs.last_ai_message (the convenience column written on run completion), not snapshotted into the outbox payload; retried deliveries re-read the freshest value and the outbox row payload stays skeleton-only. The claimed outbox row is never mutated in place. - Gateway wiring scopes the read to the outbox row's owner_user_id so a stale row cannot pull another user's run content. - Failed runs keep the error-only skeleton: a partial answer on a failed run would be misleading. Tests: 9 new worker cases (rendering incl. truncation, sync/async resolvers, failure/missing fallbacks, no resolver call for failed runs). Full scheduled + outbox + channels regression green (441 passed), ruff check + format clean. * fix(scheduler): address review feedback on notification delivery reliability Review follow-ups for the scheduled-task IM notification outbox: 1. Crash recovery for "sending" rows: claim now stamps updated_at and the worker reconciles rows stuck in "sending" past a 10-minute lease back to "pending" before every claim. Per-row try/except isolates a crashing delivery from the rest of the claimed batch. 2. Double-send race under multiple workers: claim returns exactly the rows its own UPDATE...RETURNING flipped (guarded by status='pending'), so a rival worker's concurrent claim can no longer be handed out twice. 3. Restore non-ASCII comments and drop the BOM accidentally introduced in test_scheduled_task_service.py. 4. Manual "run now" completions no longer enqueue IM notifications; the user is already watching the UI. Missing trigger metadata fails safe towards notifying. 5. Channel-outage failures no longer consume the retry budget: they are parked with count_attempt=False on a flat 15-minute backoff so an hours-long outage cannot exhaust retries and drop the notification. Tests: repository 16, worker 21, scheduled-task service 31; targeted regression across notification/scheduler/channel suites: 511 passed. * fix(scheduler): address round-2 review on IM notification delivery Park WeCom transport/outage failures without burning retries, bound worker stop, include task titles, align enqueue with delivery, and document the IM push behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(scheduler): complete IM notification docs and drop unused list API Add backend/channels/migrations AGENTS notes, CHANGELOG and config.example hints for the outbox push path, and remove the unused list_by_task_run repository method flagged in review. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(scheduler): address round-3 review on IM notification delivery Cap channel-down parking by row age and parked attempts, redact IM egress content, narrow WeCom transport exceptions, add lifespan wiring tests, and run ruff format so lint-backend passes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(scheduler): redact entire PEM blocks in IM egress text The previous BEGIN-only pattern left key material and the END marker in notifications. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(persistence): align 0014 parked_attempts with bootstrap schema tests Backfill existing rows with a temporary server default, then drop it so create_all matches Alembic, and move HEAD anchors to 0014. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(persistence): format notification tests and shorten alembic revision id ruff format --check failed on test_notification_delivery_worker.py. 0015_notification_delivery_parked_attempts exceeds alembic_version.version_num (VARCHAR 32) and truncates on Postgres CI; use 0015_parked_attempts instead. * test: fix post-merge CI failures after migration renumber Update migration 0015 test HEAD expectation to 0017_parked_attempts and add queue_timeout_seconds to the lifespan notification worker mock scheduler config. * fix(persistence): chain the notification outbox migrations after 0026 Main's head moved from 0016 to 0026_mcp_task_lease_tokens, so the two outbox revisions are renumbered and re-parented: - 0017_notification_deliveries -> 0027_notification_deliveries - 0018_parked_attempts -> 0028_parked_attempts Only the revision ids, parents and the migration guide change. * fix(scheduler): adapt the notification hook and its docs to main - tests: the completion tests assert on the atomic complete_run record; new cases cover a completion that was not recorded, a task deleted meanwhile, a failed task read, and the outbox being off. - AGENTS.md: drop the root note and shorten the backend note to a pointer; the two notes pushed two effective guidance chains past the 96 KiB limit. - CHANGELOG: move the entry from the released 2.1.0 section to Unreleased and add its link references. * test(persistence): move the chain-head pin to the outbox migrations test_migration_0026 pinned 0026 as the head, so it failed once the outbox revisions chained after it. It now checks chain membership like the older migration tests, and a new test file owns the head pin, checks the parents and the 32-character revision id limit, and runs the upgrade and downgrade. * fix(gateway): wire scheduled-task notifications after the channel service starts The outbox wiring sat in the scheduler block and asked for the running channel service there. #5035 moved the channel service start to after the scheduler, and the two changes merged without a conflict, so on every startup the lookup returned None and the feature disabled itself with the "write-only outbox" warning. The lifespan tests patched the lookup to a constant and kept passing. The wiring now runs in its own step after the channel service start. The scheduler is built as on main, and ScheduledTaskService gains attach_notification_outbox() so the enqueue side is enabled once the delivery worker is running. The guard, the worker and the shutdown order are unchanged. Tests: the lifespan helper can model the real accessor (None until the channel service has started); a new case fails without this fix. * fix(scheduler): enqueue from scheduler start and keep the push when the title read fails Follow-up to the startup-order fix, from review. - The enqueue side is wired with the scheduler again (it only needs the durable table), and only the delivery worker waits for the channel service. Attaching it later left a window after the scheduler started in which a finished run wrote no outbox row. If the channel service is missing or the worker cannot start, the enqueue side is detached, so the outbox still never becomes write-only. - A failed task read after complete_run no longer drops the notification: the title is optional and the message falls back to the task id. A task deleted meanwhile still stays silent. * fix(channels): park WeCom notifications while the SDK socket is closed The WeCom SDK raises a plain RuntimeError when its socket is not open, and send_notification only mapped ConnectionError, TimeoutError and OSError to ChannelUnavailable. A disconnect therefore spent the counted retry budget and settled the row as failed after about 15 minutes, although the docs say transport outages park without consuming retries. send_notification now asks the client's is_connected first. The new test uses a real, never-connected SDK client: it raised RuntimeError after three retries before, and parks now. * docs(scheduler): describe which outcomes notify; mirror the changelog entry - README (en/zh) and CONFIGURATION.md: only runs that finish as success or failed notify; occurrences that end without a finished run (launch error, queue timeout, restart recovery) do not. Parking lasts up to about a day. - CHANGELOG: cite the issue as text like the rest of the file, and add the entry and link reference to CHANGELOG_zh.md. - AGENTS.md: drop the backend pointer so this branch adds nothing to the guidance chains closest to their size limit; the scheduler-side rules and the lifespan order now live in app/channels/AGENTS.md. * docs(changelog): cite #6135 next to #4843 for the notification outbox entry * fix(scheduler,channels): re-check the IM binding at delivery time; park WeCom transport errors, count its rejections Review round 1 on #6135. The delivery worker now takes resolve_connections (the Gateway wires ChannelConnectionRepository.list_connections) and, before the channel liveness check and the summary read, requires the row's owner to still hold a connected (provider, external_account_id). A disconnected target is finalized with mark_failed(terminal=True); a lookup failure or an unexpected result shape parks the row. Terminal outbox rows ignore later status writes. WeCom: the SDK raises RuntimeError from send_message for a socket that is not open and for a dropped connection, but also for a platform rejection ("Reply ack error: errcode=..."). Only that prefix keeps counting; other RuntimeErrors, ConnectionError/TimeoutError/OSError and websockets' ConnectionClosed park the row as ChannelUnavailable. * fix(channels): treat a WeCom ACK rejection as proof the transport is up; pin the rejection prefix through the retry branch and the real connection-row shape --------- Co-authored-by: GaoTianyuanhr <75788039+GaoTianyuanhr@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: GaoTianyuanhr <GaoTianyuanhr@users.noreply.github.com> Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>	2026-10-01 17:59:30 +08:00
huyengiang101086-spec	baae72d876	fix(channels): point the Discord missing-dependency hint at the discord extra (#6054) ... * test(scripts): cover discord extra auto-detection The discord mapping in scripts/detect_uv_extras.py worked but was neither listed in the "currently maps" docstring nor covered by a test, unlike the buzz mapping that grew up beside it. * fix(channels): point the Discord missing-dependency hint at the discord extra discord.py is the only IM transport that is not a core dependency — it ships in the optional `discord` extra, which detect_uv_extras.py, backend/Dockerfile and deploy.sh already expand. Its ImportError hint told users to run `uv add discord.py` instead, which rewrites pyproject.toml/uv.lock in a tree every restart syncs with `--locked`. Mirror the buzz hint wording and note the extra in config.example.yaml next to the buzz block. --------- Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com>	2026-10-01 10:51:37 +08:00
Grapette.L	0110845f66	fix(config): guard the recovered stream cleanup delay like its heartbeat sibling (#5850)	2026-09-30 19:26:50 +08:00
Hyeonsang Cho	b37f769b67	fix(persistence): reject SQLite file: URIs instead of opening them as filenames (#6069) ... * fix(persistence): reject SQLite file: URIs instead of opening them as filenames resolve_sqlite_conn_str passed a file: connection string through unchanged, but LangGraph's SQLite checkpointer and Store factories (SqliteSaver, SqliteStore, AsyncSqliteSaver, AsyncSqliteStore) connect without uri=True, so SQLite treated the whole URI as a literal filename: file:checkpoints.db?mode=rwc created a file with that exact name in the working directory, file::memory:?cache=shared persisted to disk, and a file:/// URI failed to open. The readiness probe did open URIs with uri=True, so it checked a different file than the runtime used and reported in-memory URIs as not_configured. Reject file: URIs in resolve_sqlite_conn_str, the one helper all four factories go through, with an error that names checkpointer.connection_string. The probe drops its URI parsing and reports a rejected connection string as unreachable. * docs(changelog): reference #6069 * test(persistence): pin the case-sensitive file: URI check	2026-09-30 09:40:02 +08:00
Stellar鱼	c917c2cc8f	fix(sandbox): reject non-finite OpenSandbox timeouts (#5944)	2026-09-28 15:16:16 +08:00
lekt8	f889ce7517	feat(community): add Unbrowse web fetch provider (#5981) ... * feat(community): add Unbrowse web fetch provider Add a community web_fetch provider backed by Unbrowse (https://unbrowse.ai), a hosted service that returns a page as markdown, over plain HTTP when that is enough and through a cloud browser when the page needs JavaScript. The tool sends one JSON-RPC tools/call POST (unbrowse.scrape) to Unbrowse's MCP endpoint, so it needs no extra Python package and no MCP client (uses httpx, already a dependency). Unbrowse is not a general web search engine, so only web_fetch is added. Changes: - backend/packages/harness/deerflow/community/unbrowse/{__init__,tools}.py web_fetch_tool with the same output shape as the Sofya, Exa and Firecrawl fetch tools (# <title> plus markdown, 4KB cap). API key is read from the config.yaml api_key field or the UNBROWSE_API_KEY env var. Optional render: auto|never|always; unknown values log a warning and fall back to auto. JSON-RPC errors, isError tool results, HTTP errors and malformed payloads come back as "Error: ..." strings. - backend/tests/test_unbrowse_tools.py: key resolution, render handling, request shape, structuredContent vs text payloads, error paths. - scripts/doctor.py, scripts/wizard/providers.py: register the key and the web_fetch provider; backend/tests/test_doctor.py covers both doctor paths. - config.example.yaml, .env.example, backend/docs/CONFIGURATION.md, tools/AGENTS.md, EN/ZH harness/tools.mdx: document the provider. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(changelog): add Unbrowse web_fetch provider entry Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-28 15:08:49 +08:00
Zeren Wang	d700ad9edf	feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client (#5906) ... * feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client - refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its configuration: one connection-precedence rule, credential fingerprint, wire-size counting and error taxonomy, shared by the guardrail gate and the memory hooks. - feat(memory): wire `memory.prescreen` and `memory.signal_classification` into DeerMem through one injected judge hook (both off by default, `enforce` gated by the shadow evaluation). - config: add the top-level `typesafe:` block and the two judging slots to `config.example.yaml` and the Helm values, bump `config_version` to 50, and sync the Helm README example with it. - test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with per-stratum gates, censoring and the no-network baseline. Review follow-ups folded in: - `mutations_accepted` counts only facts that survive capacity enforcement and the upload-event scrub that precedes persistence, only removals/reinforcements of ids that existed, and persisted near-duplicate merges (matched by id, or by identity for legacy id-less facts); it is published only after persistence succeeds. - The cached manager hot-reloads its judge when a memory slot or the shared `typesafe` block changes (including a credential rotation); publication is serialized and revalidates the config generation, so a build started before an `enforce` -> `off` rollback cannot land after it. - Cache provenance is per answer: the `cached` flag and the reported model follow the answers a side actually consumed, across partial merges and answerless retries. - An oversized noul integer stays a per-question error instead of raising. - Incremental call savings use the shared scored denominator. - The sensitive-stratum gate requires evidence for each of identity, preference and correction. * fix(typesafe): resolve one credential per layer and refuse unusable connection values Review follow-ups on the shared TypeSafe client (#5906): - A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the shared `typesafe:` block: the first layer that sets either credential setting decides it, and inside one layer the literal key wins (the standalone semantics). A consumer whose endpoint does not accept the block's key no longer sends it on every call, and the missing-credential message keeps naming the variable that layer configured. - `base_url` is validated where it is configured: `http(s)` only, with no query, fragment, embedded credentials, whitespace or control characters, and a malformed port fails at construction instead of on every request (httpx raises `UnsupportedProtocol` / `InvalidURL` per call, and the query would have been swallowed into the appended endpoint path). - A credential that cannot travel as a header value — surrounding whitespace from a secret mounted from a file, a non-printable character — is refused when the connection resolves. It used to construct fine and then put the whole `Bearer <key>` value into h11's `LocalProtocolError` message, which the guardrail middleware's `logger.exception` prints. - The client raises transport failures without chaining the original exception, so that message can no longer carry the credential through the cause chain; the exception type stays in the message for diagnosis. - `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config` path: `max_attempts: true` was silently read as `1`. An integer stays a usable float, so `timeout: 5` keeps working. Tests: 20 cases across `tests/test_typesafe_config.py` and `tests/test_typesafe_client.py`, each red on the previous revision and green now. 367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean. Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`, `config.example.yaml`. * fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence Second review round on #5906: the remaining items from the review thread and from the follow-up pass. Typesafe client (the reply is treated as hostile input too): - Send `Accept-Encoding: identity`, refuse a body that carries a non-identity `Content-Encoding`, and read through a bounded `iter_bytes` that stops past `MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be decoded in full and, on the gate path, allowed the tool call unexamined; it is now `invalid_response` (a denial with `fail_closed: true`). - Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting escaped the module's taxonomy, so the memory consumers lost the round's record instead of recording a fallback. Memory judge: - A request-level failure is recorded as `request_failed`, never `no_verdict` (`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable endpoint was indistinguishable from a provider with no usable answer). The adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record and log; `no_verdict` stays for a question-level failure, and an ineligible side keeps its own reason even when the round's request failed. Shadow evaluation (the enablement gate reads the evidence, not one number): - The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to meet the 1% target; a sample that only meets the point estimate is `INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound). - New `baseline_incremental_savings` gate: a baseline that saves as many calls as the model — the shipped all-trivial sample — fails instead of enabling. - Negative counts (`mutations_accepted`, message / batch / token counts) are treated as absent rather than scored: `-3` read as "nothing lost", adding a phantom saving to the numerator and the denominator. Docs: the design documents cited but absent from the repository are replaced with in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`, `GUARDRAILS.md`), and the gate and limit wording now matches what the code enforces. Tests: 31 new or updated cases, each red on the previous revision (compressed body refused even when it decodes to a valid envelope, body over the cap refused, body at the cap answered, deep nesting mapped, `request_failed` on each request-level failure, the bound and baseline gates, the negative counter). 454 focused tests, `make test-blocking-io` (167) and the eval suites pass; `ruff check` and `ruff format --check` are clean. * fix(memory-eval): pair the baseline comparison on the same records Review follow-up on the incremental-savings gate added in ed930478 (P2): both sides of the comparison are now the records the no-network baseline can score (`trivial_only` plus a readable signals list). Subtracting that subset's heuristic savings from model savings over *all* scorable skips credited the model with batches the baseline never saw: 300 reviewed skips with a single annotated record computed 300-1=299 "incremental", passed the gate and set `enable_recommended`, while the paired comparison was model=1 against heuristic=1 (incremental 0). - `Evidence` carries the paired numbers (`baseline_paired_skips`, `baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one heuristic count, filled by `build_report` from the baseline summary. - The gate reports `INSUFFICIENT` when no scored skip carries the annotation and when the annotation covers only a subset of the scored skips (the message carries the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive paired gain, and `PASS` only on a positive gain over the full paired set. - Docs state the pairing/coverage requirement in `backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring. Tests: the reviewer's counterexample as a report-level regression (it failed with `assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true), a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields. 344 focused tests, `make test-blocking-io` (167) and the eval suites pass; `ruff check` / `ruff format --check` clean. * fix(typesafe): bound the sync body read by the deadline `deadline_seconds` only covered the sync path's headers, the completed body and the parse, so an endpoint that drips bytes just under the read timeout held the caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget. The read loop now checks the budget before every transport read, which bounds the overshoot to one read. The pieces are the transport's own reads, not re-chunked output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk, so a check at that granularity would never see a drip. `tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24 pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now (~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track the new granularity. * fix(memory): read the YAML off spelling, and veto a skip over the whole batch Two review findings on the memory judging path. `memory.prescreen.mode` / `memory.signal_classification.mode` rejected the `mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1 parses the unquoted value as a boolean, so both fields saw `False`. The failed reload left the previous judge running, so both fields now normalize `False` to `off` before the `Literal` runs. `True` stays a validation error (no mode it could mean). The examples quote the value, and the docs record why. The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch it answers for is the whole post-watermark feed, so an explicit correction in an earlier turn of an eight-message batch did not stop the skip that then advanced the watermark past it. `detect_signals` gains a `window` parameter (default unchanged) and the updater scans the full feed for the veto; the extraction hint and the reinforcement gate keep the window. Both regressions are red on the previous head: the config test reproduced the `input_value=False` rejection through `AppConfig.from_file`, and the batch test reproduced the prescreen judging (and skipping) a batch whose first turn carries the correction.	2026-09-27 20:28:05 +08:00
李靖	10bd065615	feat(agents): persist tool artifacts as resolvable handles (#4676) (#4929) ... * feat(agents): persist tool artifacts as resolvable handles (#4676) MCP and sandbox tools return file paths, URLs, and task ids that are lost when context compaction summarizes the conversation. This adds a per-thread artifact handle registry so the model can reference artifacts by short deterministic handles (art_xxxxxxxx) that survive compaction and resolve to real references at the tool-call boundary. - ThreadState.tool_artifacts: reducer channel capped at 100 entries, same-handle latest-wins, survives summarization compaction - artifact_registry: deterministic handle derivation from (thread_id, tool_call_id, call_index); extraction precedence structured_content -> file/image blocks -> conservative text scan; HTML-escaped model-context rendering with char budget - ArtifactCaptureMiddleware: before_model hook captures lightweight metadata (never content) and tracks handle consumption - ArtifactResolutionMiddleware: rewrites handles in tool-call args to real references before execution; unknown handles untouched - DurableContextMiddleware: projects the registry into the hidden durable-context data block ([available]/[consumed] status) - config: new tool_artifacts section (all switches default on) - frontend: artifact badges on tool-call steps keyed by tool_call_id Fixes #4676 * fix(agents): harden artifact registry after review (#4676) Review findings on #4929: - capture/consumption updates emitted in the same before_model call are now concatenated instead of dict-merged; dict-merging clobbered the shared tool_artifacts key and permanently dropped fresh captures - handles embed a per-result ordinal so multiple references inside one tool result never collide (two paths in one text block previously collapsed into one handle) - structured_content extraction pulls concrete refs from known keys (file/path/url/task_id/..., string or list-of-string values) instead of storing a truncated JSON preview, falls back to an untruncated whole-object data entry, and no longer skips file/image content blocks - plain-string tool results (e.g. bash/write_file output) are scanned for sandbox paths / remote URLs too, gated by detect_refs_in_text - wire max_entries through configure_tool_artifact_max_entries at assembly (clamped 10..1000) and gate detect_refs_in_text scans - resolution requires both enabled and resolve_handles_in_args, matching its docs - add real ToolCallRequest.override coverage and the spec 14.4 integration cycle (capture -> inject -> resolve -> consumption) * fix(agents): address second-round artifact registry review (#4676) Follow-up findings on #4929: - drop generic `output`/`outputs` from structured ref keys: prose under those keys no longer becomes a bogus file entry (falls back to the whole-object data entry; path-shaped refs are still caught by the free-text scan) - strip closing quotes, brackets, and backticks from detected refs so markdown- and JSON-wrapped paths resolve to clean real references - remove the process-wide cap setter: the reducer enforces only an absolute ceiling (1000) and ArtifactCaptureMiddleware applies the configured max_entries per agent when emitting updates, so agents with different configs in one process never share or flap a cap - skip extraction for tool messages whose tool_call_id already has registry entries and memoize known-empty results (bounded FIFO), so steady-state capture cost drops to the new message tail instead of a full-history rescan per model call - correct the inverted ordering claim: capture is position-independent in the wrap chain because it is a before_model hook reading state messages; fixed the assembly comment, AGENTS.md item 15, and the ordering test docstring CI parity verified locally: ruff check + format --check, full offline suite, blocking-io suite, collect-only, uv lock --check, check_agent_guidance, frontend format/lint/typecheck/build/tests. * fix(agents): sliding-window cap and URI gates for artifact registry (#4676) Third-round review findings on #4929: - max_entries is now a sliding retention window: fresh captures are always registered, and when the projection exceeds the configured cap the middleware appends a trailing {"op": "trim_to", "keep": N} directive that the reducer applies by evicting the oldest entries. The previous remaining-budget emission froze the registry at the cap, permanently dropping post-compaction artifacts — the exact failure mode #4676 targets — and silently disabled the round-2 rescan optimizations at the point threads grew longest - _track_consumption memoizes settled tool-call args in a bounded FIFO alongside the existing empty-result memo, so historical AIMessage args are not regex-rescanned every model call - file/image content-block URLs must be http(s) or path-shaped; data:/blob: URIs (MCP embedded resources carrying base64 payloads) can no longer enter thread state, tool args, or crowd out the render budget; protocol-relative //host forms are rejected as well - structured-content ref values go through the same URI-shape gate as content blocks, closing the embedded-payload hole via known keys such as url/urls - tool_artifacts.enabled now gates the whole middleware including consumption tracking via an early return in before_model Docs updated to describe the sliding window, URI gates, enabled scope, and both bounded memos. * fix(agents): final evictions and pending-aware consumption for artifact registry (#4676) Fourth-round review findings on #4929: - the sliding window interacted with the seen_call_ids skip to resurrect evicted entries every round: once an entry was trimmed, its tool_call_id left the seen set while its ToolMessage stayed in context, so the next model call re-extracted it, re-registered it as newest, and evicted something else — a never-converging rotation. The empty-result memo is now a unified _processed_results memo that records every extraction regardless of outcome, making evictions final; steady state over a saturated thread is quiescent (pinned by a multi-round test asserting no updates and zero extractions) - the consumption scan built its handle map from pre-capture state, so a call referencing a handle captured in the same before_model call missed, got settled as quiet, and its consumption mark was lost forever. _track_consumption now receives the pending capture entries and resolves against state + pending; a call settles as quiet only after every referenced handle resolved, so transient misses retry next round instead of permanently suppressing the mark AGENTS.md entry 15 updated to describe both memos and the eviction-is-final semantics. * fix(artifacts): address occurrence identity, PII and URL review findings * fix(artifacts): enforce resolved policy checks and bounded registry handling --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-27 11:09:41 +08:00
xiaodu55	64e499be98	feat(memory): redact queued memory payloads (pii_redaction follow-up) (#5577) ... * feat(memory): redact queued memory payloads (pii_redaction follow-up) Closes the remaining vector from #3190 (memory storage): request-scoped redaction leaves originals in thread state, and MemoryMiddleware queued that raw conversation into the durable extraction buffer — so the extraction model's input and the persisted facts carried raw PII even with pii_redaction.enabled. - MemoryMiddleware gains pii_redaction_config (same pattern as summarization / durable-context / title); the payload returned by _resolve_add_args() is redacted at the enqueue boundary before add()/aadd(), covering both after_agent and aafter_agent - Reuses the shared redact_text() seam from #5527 via active_pii_detectors/_Redactor/_redact_content; the buffered payload itself becomes clean, not just the extraction model's view - Wired at the lead-agent assembly site - Tests: queue-payload redaction enabled/disabled, detector toggles, message immutability, token stability across turns * fix(memory): reserve existing placeholder indices across the queued batch Review round 1 on #5577: thread state mixes raw user turns with already-redacted tool results and summaries, so a token in a later message must not collide with a new value in an earlier one. Pre-scan the whole queued batch with redactor.reserve() before allocating. Also extend the two MemoryMiddleware stubs in test_lead_agent_model_resolution.py to accept the new pii_redaction_config kwarg (CI backend-unit-tests red). * fix(memory): redact the compaction-triggered memory_flush_hook queue too Review round 2 on #5577: with memory + summarization enabled, memory_flush_hook() queues list(event.messages_to_summarize) directly into MemoryManager.add_nowait(), bypassing MemoryMiddleware entirely — and compaction removes those messages from state right after, so the after-agent redaction can never repair a raw batch queued there. Extract the batch redaction (reserve pre-scan + rebuild) into a shared redact_queued_messages() used by both MemoryMiddleware's enqueue boundary and the hook; the hook gains an optional pii_redaction_config wired via functools.partial at the summarization factory (subagents use skip_memory_flush=True and are unaffected). Regression covers the hook path enabled/disabled/absent. * test(summarization): unwrap the partial in the factory hook identity check edd916e wrapped memory_flush_hook in functools.partial to carry the pii_redaction config, but missed committing the matching update to test_factory_attaches_memory_flush_hook_by_default — a functools.partial is not equal to its wrapped function, so the identity assertion failed deterministically (caught by static review on #5577). * fix(memory): cover tool-call arguments and lenient text blocks in queued payloads Review round 4 on #5577 — two reproduced gaps still allowed raw PII into memory payloads with pii_redaction.enabled=true: - OpenViking retains the full message object, so parsed tool_calls args and provider-format arguments in additional_kwargs bypassed the content-only redaction; queued payloads now redact tool-call args recursively (and reserve them in the pre-scan). - DeerMem's format_conversation_for_update reads p.get("text") of any dict block, not only type=="text" — _redact_content now redacts every string-valued text field so a lenient consumer cannot bypass it. Both repro shapes are pinned as regressions. * fix(pii): value-derived placeholders for stable capture identity Review round 5 on #5577: sequential allocation is order-dependent — the same unchanged message re-redacted in a later batch can get a different token, which flips OpenViking's content-signature and replays already recorded turns into persistent memory sessions. Placeholders are now a pure function of the matched value (sha256-6 per category): identities stay stable across batches, enqueues, seams, and backends with no shared state and no stored mapping. - _Redactor becomes stateless; the reserve machinery (index pre-scans, _PLACEHOLDER_PATTERN) is removed along with its call sites - redact_texts keeps its signature, now a simple per-text redaction - reserve-semantics tests are superseded by cross-enqueue identity tests (two successive captures where the second batch carries higher existing placeholders); the compiled-graph echo regex matches the new format * fix(pii): 128-bit placeholder digests; redact tool-argument keys Review round 6 on #5577: - [P2] The 24-bit truncation collided distinct identities (reproduced: contact3513@ / contact3727@example.com both rendered [EMAIL_575dbb]). Retain 128 bits of the digest so separate people stay separate at any realistic volume; the compiled-graph echo regex and the AGENTS.md entry follow the new format. - [P2] _redact_strings redacted values but preserved mapping keys, so {"contacts": {"alice@example.com": "manager"}} kept the raw email as a key. Redact string keys along with values; deterministic 128-bit tokens make distinct-key collisions a hash-collision non-event, and a redacted key equal to another key merges the entries. Regression pins the email-keyed arguments shape. * docs(agents): trim PiiRedactionMiddleware entry; drop stale reserve wording The chain grew past the 98304-byte AG002 hard limit again (main growth plus the round-6 format example). Trim the entry to fit, and replace the stale reserve-indices wording with the value-derived semantics that replaced it. * docs(agents): shrink PiiRedactionMiddleware entry below the chain budget main moved (#5609) and the base comparison flipped the nearly-over chain back to an AG002 error (98441 > 98304). Trim the entry to the facts that are not documented elsewhere: seams, allowlist, order, enqueue boundary. * fix(pii): letters-only value-derived tokens; wire the factory SDK path Review round 7 on #5577: - [P2] Hex digests carry digit runs that later digit-anchored detectors re-scan, corrupting minted tokens into nested placeholders (reproduced: user280@example.com -> [EMAIL_…[PHONE_…]…], ~1/140 emails). The 128-bit digest is now encoded over a-z, so a minted token survives the full pinned detector order. - [P2] create_deerflow_agent (the SDK path) constructed MemoryMiddleware and DurableContextMiddleware without pii_redaction_config; the knob is now an explicit optional parameter threaded through _assemble_from_features (the factory reads no config files by design, so SDK callers pass it explicitly). - [Nit] hoist the Command rebuild comprehension out of the per-message loop in _redact_result. - [Nit] drop the dead _iter_tool_calls helper and Iterator import. * test(factory): pin pii_redaction_config wiring; document the parameter Review round 8 on #5577: - [P3] test_create_deerflow_agent.py had zero coverage of the pii_redaction_config parameter; two regressions now assert that create_deerflow_agent(pii_redaction_config=cfg) builds a MemoryMiddleware and a DurableContextMiddleware carrying cfg, and that the default is None (redaction off). - [Nit] document the parameter in the create_deerflow_agent docstring: None leaves redaction off; a middleware= takeover bypasses the assembly that would apply it. * fix(pii): HMAC-scoped placeholder tokens; install model-bound redaction in the SDK chain Review round 10 on #5577: - [P2] sha256(category||value) is an unkeyed, globally deterministic fingerprint: a provider or memory backend can confirm candidate values offline and link identical values across users and threads, contradicting the documented no-re-identification guarantee. Placeholders are now HMAC-derived over a deployment-scoped token_secret (new config field); the docstring documents the linkability scope for both modes. - [P2] _assemble_from_features() never appended PiiRedactionMiddleware, so the SDK knob enabled only the downstream seams while the primary model still received raw content. The middleware is now appended in the same relative position as the lead-agent base chain when the gate is on, with a model-call regression through the assembled SDK chain. * docs(pii): document token_secret; update the placeholder format examples Follow-up to the round-10 review on #5577: - config.example.yaml documents the new token_secret knob: a deployment-scoped secret makes placeholders HMAC-derived and only linkable within one deployment; leaving it unset keeps publicly computable fingerprints (documented trade). - the PiiRedactionConfig class docstring's placeholder example now matches the shipped format (128-bit HMAC digests over base-26 letters). * docs(pii): update the module docstring's placeholder format example The module docstring still showed the sequential [EMAIL_1] example; align it with the shipped value-derived format (128-bit HMAC digests over base-26 letters). * fix(pii): require a non-empty token_secret when redaction is enabled Review round 11 on #5577: with the secret optional, the default enabled configuration minted empty-key HMAC digests — publicly computable, globally linkable fingerprints of the raw values, confirmable by offline guessing for low-entropy identifiers such as phone numbers. - PiiRedactionConfig now rejects enabled=true without a non-empty token_secret (minimum 16 characters); the unkeyed mode is gone entirely. - The request-scoped redactor in _process_request now carries the configured key: it built its redactor without it, so user-message placeholders silently diverged from every other seam for the same value. The empty-key guard in _placeholder_token is the backstop that keeps every path keyed. - Module/config docstrings and config.example.yaml no longer describe the placeholders as irreversible or the secret as optional; the stale reserve-semantics wording and the "memory path not covered" note are corrected. - Regressions: validation errors for missing/blank/short secrets, and the user-message boundary minting the same keyed token as redact_text. * fix(memory): redact original_user_content on queued memory payloads Review round 12 on #5577: UploadsMiddleware preserves the raw user turn in additional_kwargs.original_user_content, but redact_queued_messages rewrote only content, tool_calls, and provider-format tool_calls — the queued copy still carried the raw value, so the memory queue and backend received raw PII with redaction enabled. The shared helper now redacts the provenance string field on the queued copy (same batch redactor, merged with the existing additional_kwargs rewrite, rebuilt via model_copy — thread-state messages stay untouched), covering both the enqueue boundary and the compaction flush hook. Regressions pin the redacted provenance field through both paths and the untouched original. * fix(memory): redact human_input_response on queued memory payloads Review round 13 on #5577: clarification replies keep the raw user answer in additional_kwargs.human_input_response.value (DeerMem reads that mapping via its read_human_input_response mirror), but redact_queued_messages rewrote only content, tool_calls, and original_user_content — the queued copy still carried the raw value into the memory queue and backend. The shared helper now rebuilds that mapping with value redacted (same batch redactor) while preserving the protocol fields (version/kind/source/ request_id/option_id/response_kind), merged with the existing additional_kwargs rewrite and rebuilt via model_copy so thread-state messages stay untouched. Regressions pin the redacted value, preserved protocol fields, the untouched original, and the flush-hook path. * chore(config): bump config_version to 47; generate token_secret on upgrade Review round 13 on #5577: existing deployments can legitimately sit at config_version 46 with pii_redaction.enabled: true and no token_secret; the mandatory-secret validator would fail their startup while config.example.yaml still said 46, so the upgrade checker considered them current and emitted no migration warning. - Bump config_version to 47 so the outdated-config warning fires for v46. - Add a v47 migration to scripts/config-upgrade.sh: when pii_redaction is enabled and token_secret is missing/blank, generate a random deployment- scoped secret (secrets.token_urlsafe) and persist it, so the upgraded deployment starts under the mandatory validation. - Regressions: a v46 enabled config is flagged outdated against the real example; the upgrade run generates a >= 16-char secret and the result passes AppConfig validation; a disabled config gets no secret. - Also switch the two v41 script tests from a hardcoded "bash" to SCRIPT_BASH/skipif, matching the documented Windows constraint and the neighboring tests (hardcoded bash resolves to the WSL stub there). * chore(helm): sync chart config_version to 47 Review round 14 on #5577: check_config_version.sh fails whenever the chart's embedded config_version lags config.example.yaml, and the chart workflow is triggered by config.example.yaml — so the v47 bump left this head failing that check. Update the values.yaml config block and the README example to 47; scripts/check_config_version.sh now passes. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-25 20:05:36 +08:00
yang rui	c814943215	feat(extensions): grant bounded host model invocation (#5796) ... * feat(extensions): grant bounded host model invocation * fix(extensions): retain model budgets and isolate schema validation * fix(extensions): normalize provider-origin model cancellation --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-25 15:24:09 +08:00
XuRuuuy	b2a2fb757a	fix(browserless): read reject_resource_types / reject_request_pattern from config (#5719) ... * fix(browserless): read reject_resource_types / reject_request_pattern from config web_fetch_tool declared both parameters, initialised them to None, and passed them to the client -- but never read them from tool config, so they were always None on the wire. BrowserlessClient implements them (and is tested for it at `test_fetch_html_with_reject_params`), so the provider's resource blocking was unreachable without patching the source. Skipping images/media/fonts is the main lever for render latency and bandwidth against a self-hosted Browserless, and it could not be turned on from config.yaml. Read both keys through a new `_as_str_list` helper that accepts a YAML list or a comma-separated string and yields None for anything unusable, so an unparseable value omits the parameter instead of reaching the server. Both keys are now documented on the Browserless web_fetch entry in config.example.yaml. Default behaviour is unchanged: an unset key is still None, so the payload is identical to before for every existing config. Adds four regression tests covering the list form, the comma-separated form, the unset case, and unusable values. * fix(browserless): validate reject list entries --------- Co-authored-by: RXQ6 <RXQ6@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-25 15:15:32 +08:00
Zeren Wang	82671596f3	feat(authz): add plugin resource targets, provenance, action and management enforcement (#5842) ... * feat(authz): add plugin resource targets, provenance, action and management enforcement PR1 of the extension-tool-authorization series (docs/plans/2026-09-24-extension-tool-authorization-spec.md): plugin authorization plumbing, no new author-facing declaration. - `authz/plugin_targets.py` is the single encoder/validator for composite targets (`namespace/part`), with one validating composer behind three kind-checked constructors; invalid input raises instead of degrading. - `authz/plugin_authz.py` is the harness decision layer (`plugin_action`, `plugin_management`, plus the batch filters the page slice will consume): disabled is a no-op, an explicit deny raises, and provider errors follow `fail_closed` like the sandbox gate. - `POST /api/plugins/{ns}/actions/{name}` decides `plugin_action:invoke` after the action resolves and before the body is streamed. - `deerflow_extension_api.auth` publishes `require_plugin_management` / `arequire_plugin_management` with two `app.state` resolver keys; the Gateway installs both. Contract version 0.2.3 -> 0.2.4. - `tools/tool_provenance.py` records host-written plugin provenance for display (assembly descriptor, Layer-2 context later); no decision reads a label. - Plugin providers use a `(config signature, loop key)` cache so a loop-affine provider is never handed to another loop or to a sync caller. - RBAC gains `plugin_actions` / `plugin_management` aliases; `config_version` 47 -> 48 across the three mirrors. Tests: new plugin target/provenance/action/management suites plus a blocking-IO anchor whose offloads were reverted to prove red->green. * fix(authz): decide plugin resources from the request's config snapshot Review round 1 on PR #5842 (willem-bd). P1 — the decision layer no longer reloads the configuration. The six `authz/plugin_authz.py` functions take a required `app_config` snapshot (the same request-scoped config the Gateway used to resolve the provider), and `resolve_/aresolve_plugin_authorization` return it as a third element, so `enabled`/`fail_closed` can never come from a second read that drifted from the one that produced the provider. A snapshot the caller could not read at all is *unavailable*, not disabled: it fails closed as `authz.config_unavailable`. The Gateway's `_plugin_app_config` now separates an absent `config.yaml` (no gate, today's behavior) from a config that exists but cannot be read or validated (deny) instead of swallowing both through `safe_app_config`. P2 — `_validated_decision` requires `type(decision.allow) is bool` on both the sync `authorize` and async `aauthorize` paths, so a provider returning `AuthzDecision(allow="false")` is a malformed verdict that follows `fail_closed` instead of being read as an allow. Both fixes are covered by regression tests that were verified red against the pre-fix behavior (P1: 200 vs 403 and resolver True vs False; P2: 12 cases across fail_closed/fail_open and sync/async), plus tests pinning that an absent config and `authorization.enabled: false` keep today's behavior. * fix(authz): treat a request-time config absence as unavailable Review round 2 on PR #5842 (willem-bd). P1 — `_plugin_app_config[_async]` no longer maps a missing `config.yaml` to `None` ("no gate"). A Gateway cannot have started without a readable config (`app/gateway/app.py` reads it through this same accessor in `lifespan`), so absence *during a request* means the configuration this process was running on became unavailable — a hot reload, an atomic replace, a ConfigMap remount. It is unavailable, not disabled: the error propagates, `resolve_/aresolve_plugin_authorization` raise `_PluginAuthorizationUnavailable(fail_closed=True)`, the action dispatcher returns 403 and the management resolvers answer `False`. The dead `(None, None, None)` allow path is gone, the resolvers return a non-optional `AppConfig` as their third element, and callers key "authorization disabled" on `provider is None` alone. This matches the rule the decision layer already documented (`authz.config_unavailable`) and the public docs. P2 — `_deny_reason_code` is total: a `reasons` that is not iterable degrades the code to `authz.denied` instead of raising `TypeError` outside the guarded provider path, which turned a denial into an unhandled error. It is deliberately not folded into `_validated_decision`: `allow` was a real `bool`, so the verdict is a valid denial, and treating it as malformed would flip it to an allow under `fail_closed: false`. Both fixes are covered by regression tests verified red against the pre-fix behavior (P1: resolver True vs False, async `PermissionError` not raised, route 200 vs 403; P2: `TypeError: 'NoneType' object is not iterable` at `plugin_authz.py:113`). The action-route tests in `tests/test_plugin_contributions.py` now install a readable `authorization.enabled: false` config instead of relying on a config-less process, and `docs/full-stack-plugins.md` plus the PR1 decision log record the new rule. * fix(authz): apply the running policy when the plugin config is unreadable Follow-up to f03d2ea9 (review round 2 on PR #5842, willem-bd). The first attempt read the P1 finding as "absence is never disabled" and failed every config read closed unconditionally. That is too broad for this repo: a config-less host is deliberately treated as having no policy to apply (`_get_route_authorization_config` falls back to a disabled config, `sandbox_authz.safe_app_config` documents the same no-op for "CI runners and direct-call tests"), and the rule turned the config-less e2e plugin host (`extension_test_fixtures.bookmark_plugin_gateway`) into a 403 — the three `bookmark-plugin.spec.ts` cases lost the save action. Reproduced locally: `POST /api/plugins/community.bookmarks/actions/save` against that fixture host returned 403 before, 200 after this change. The rule is now "has this process loaded a configuration?", not "was the last read successful?": - No loaded config (`peek_loaded_app_config()`, new read-only accessor on `deerflow.config.app_config`) and no `config.yaml`: no policy to apply, so the gate stays a no-op — unchanged for CI runners, direct-call tests and hosts that mount only the plugins router. - A loaded config whose file disappears or cannot be read: *unavailable*, never *disabled*. The resolvers raise `_PluginAuthorizationUnavailable` carrying the failure flag of the configuration this process runs on (authorization disabled there -> no gate; enabled -> that policy's `fail_closed`, `True` by default), so an enabled Gateway answers 403/False in the window instead of letting protected plugin actions and management routes through. - A config file that exists but cannot be read or validated right now: fail closed when nothing is loaded (round 1's rule), otherwise the loaded policy decides. `peek_loaded_app_config()` is preferred over a Gateway-side startup snapshot: it reflects the last successfully loaded config, so a hot-reloaded policy is what applies during the window, and there is no lifecycle hook to keep in sync. Evidence: the three no-config/lost-config branches in `tests/test_plugin_management_guard.py` and `tests/test_plugin_action_authorization.py`, round 1's unreadable-config tests unchanged, `test_peek_loaded_app_config_survives_a_missing_file` in `tests/test_app_config_reload.py`, and the fixture-host probe above. Docs (`docs/full-stack-plugins.md`, `reference.mdx` en/zh) and the PR1 decision log state the final rule; the log records the superseded variant. * fix(authz): keep a denial whose reasons raise while being read * fix(authz): intersect plugin batch filters with the candidates The plugin_page/plugin_management batch helpers returned the provider's `filter_resources` answer verbatim, so a custom provider could hand the projections a target that was never asked about ("*", another namespace, a phantom page). Every other filter call site re-projects from its own candidate list; do the same here, state the intersection on the provider contract, and pin it with a regression test that fails on the old pass-through. * docs(plugins): map a management denial to 403 in the route example The contributed-route example called `arequire_plugin_management` and let the documented `PermissionError` escape. The Gateway installs no `PermissionError` handler, so a route copied from the example answered 500 on a normal policy denial instead of 403. Both example routes now translate the denial, and the surrounding bullets say why the translation is the route's job. * docs(extensions): point the 0.2.4 contract row at the PR that ships it Every other row in the version-history table links the pull request that shipped the version; 0.2.4 linked the authorization tracking issue instead. Both the English and Chinese tables now link this PR.	2026-09-25 14:46:12 +08:00
yang rui	1da284bb3a	feat(prompts): add operator overlays for core system prompts (#5794) ... * feat(prompts): add operator overlays for core system prompts * test(prompts): isolate SOUL forwarding test from operator config * fix(prompts): wrap complete subagent system message with overlay * fix(subagents): serialize prompt overlays in durable batches --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-25 09:51:11 +08:00
Nan Gao	bcaf6db36a	feat(models): add declarative reasoning capability contract (#5780) ... * feat(models): add declarative reasoning capability contract (#5073) Introduce an optional per-model `reasoning:` block beside the legacy `supports_thinking` / `supports_reasoning_effort` booleans. It declares thinking availability (unsupported | optional | required), the accepted effort values with aliases, default, and serialization path, the payload dialect, and the reasoning-history requirement. When present the booleans are derived from it and contradictory profiles fail at config load. `deerflow.models.reasoning` normalizes legacy and declared profiles into one immutable contract and resolves every caller's generic request against it. `create_chat_model` is the single enforcement point: required-thinking models never receive a synthesized disable payload, unsupported effort values never reach the provider, and the dialect synthesizes the on/off payload when no template exists. Legacy profiles keep the historical path byte-for-byte. The lead agent and subagent descriptor resolve the same policy first so run metadata reports effective values. `/api/models` and `DeerFlowClient` project the contract as `reasoning`; the frontend derives mode clamping and effort choices from it through `core/models/reasoning.ts`, and user preferences accept provider-specific effort tokens. The setup wizard's Z.AI GLM-5.3-Flash profile migrates to the contract, restoring its low/high/max effort control. * docs(models): move the reasoning contract guidance into models/AGENTS.md backend/AGENTS.md sits within 46 bytes of the 96 KiB effective-chain hard limit for the middlewares, runtime, sandbox, and subagents guides, so the new section pushed scripts/check_agent_guidance.py over it. The models guide is the nearest file for the factory and the contract module, and its chain has ample room. * fix(models): address review on the reasoning contract - vLLM dialect: the enable path now mirrors whichever chat_template_kwargs switch the template declares, like the disable path already did, so a template using the older `thinking` key no longer also grows `enable_thinking`. - Validate operator-supplied effort values at `effort.path` inside `when_thinking_enabled`, `when_thinking_disabled`, and the `thinking` shortcut against the contract, not only the profile-level value; they are forwarded when the caller chooses nothing, so the "unknown values never reach the provider" guarantee now holds for them too. - Constrain `effort.path` to dotted identifiers and reject single-segment paths that would replace a whole mapping (`extra_body`, `thinking`, `model_kwargs`, `default_headers`, `default_query`). - Z.AI GLM-5.3-Flash profile: `default: high` instead of `max`, because the default also governs summarization, title generation, and subagent calls that never choose an effort; `max` stays selectable. Documented in the config example, the configuration manual, and the design note. * fix(models): close reasoning contract review gaps * fix(models): preserve native Ollama reasoning config * fix(models): keep native reasoning strings and project reasoning into the descriptor - `ModelConfig.reasoning` also accepts the level string ChatOllama takes for gpt-oss style models (`low` / `medium` / `high`); the factory forwards both the boolean and the string on the legacy path, so profiles that passed `reasoning: high` through before the contract field existed keep loading and keep sending it. - The assembly descriptor no longer excludes `reasoning` from `model_parameters`: the native ChatOllama value and a declared contract's dialect/history/effort change the request payload, so they now move the assembly fingerprint. - e2e: the sidecar spec pinned the pre-contract body where a model without effort control still received `reasoning_effort: minimal`; the preset is now dropped for such models (the Gateway discarded it anyway), so the assertion expects the key to be absent.	2026-09-24 20:39:30 +08:00
Zeren Wang	577c39643b	feat(guardrails): add TypeSafe (Jev) pre-execution tool risk gate (#5712) ... * feat(guardrails): add TypeSafe (Jev) pre-execution tool risk gate Adds TypeSafeGuardrailProvider, a network guardrail provider that asks one `noul` question per probed tool call and denies calls whose Jev risk probability reaches `threshold`. It reuses the existing GuardrailProvider slot: no middleware, config-schema or dependency changes. Semantics that are load-bearing: - state is strict JSON (no `default=`, `allow_nan=False`, string keys only); unserializable arguments, lone surrogates among them, and argument text above `max_state_chars` are denied locally with zero network requests, never truncated and sent. - the async path cancels in-flight requests through `asyncio.timeout`; the sync path checks the deadline after the response headers arrive, around the body read and before accepting the result, and drops late results. No total return-time bound is promised on the sync path. - clients are created per evaluation inside a context manager, and the injected `transport_factory` is called once per network-needing evaluation: httpx closes an injected transport with its client, so a shared instance would be dead on the second call. - denial reason messages carry probability, threshold, served model version and state digest, so the threshold comparison can be replayed from the run journal; local denials record no probability or model version, and allow decisions stay unpersisted (the audit path is unchanged). - provider errors carry no response body or arguments: malformed responses report the field and the offending type only, so a response echoing tool arguments cannot reach middleware logs or the evaluation report. The online evaluation harness (`backend/scripts/eval_typesafe_risk_gate.py` plus its labeled case set) is the threshold/tools evidence that must precede enabling interception. It reports four mutually exclusive populations (uncached network, cache hits, unprobed tools, local refusals), separates model classification from local refusals and fail-closed errors, measures latency from the caller entering `aevaluate` through cleanup, discloses censored samples, probes DNS/TCP/TLS separately, and flags unprobed tools as a coverage gap rather than a pass. Real responses are never written to disk. Cache-pass samples are classified from `decision.metadata['cached']` and counted once, warming calls are retained as an auxiliary group with their failures named in the report and JSON, and an optional connection-probe diagnostic can never discard completed evaluation results. Tests: the provider suite over MockTransport plus the middleware integration, a blocking-IO anchor that drives `aevaluate` against a real loopback HTTP server with a meta-check proving the sync path on the loop trips the gate, and harness cases covering the reporting defects. Every fix above was verified to fail its new tests when reverted. Docs: GUARDRAILS.md gains Option 4 (privacy, local-refusal trade-off, deadline semantics, audit boundary, coverage/bypass risk, Allowlist->RBAC migration, lone-surrogate local refusals and the no-response-body property of error messages) and config.example.yaml gains a smoke-scope example. Config: the new example block bumps `config_version` 46 -> 47, with the Helm chart's embedded version and README example kept in step as `scripts/check_config_version.sh` (chart.yaml CI) requires. * feat(guardrails): whitelist tool calls inside the TypeSafe risk gate Static allow rules now live in the provider instead of waiting on a composite provider. `whitelist` mirrors `AllowlistProvider.allowed_tools`: omitted means "not enforced", `[]` refuses every tool, and a list refuses every tool outside it locally (`typesafe.tool_not_whitelisted`) before state construction, caching and networking, independently of `fail_closed`. Whitelisted tools are still probed and still denied when their risk probability reaches `threshold`, and the whitelist is checked before the probe scope so an unlisted tool cannot inherit an allow from being unprobed. Deny rules stay with `authorization.*`. The policy version moves to 1.1.0, the evaluation script reports whitelist refusals as their own population, and `GUARDRAILS.md` / `config.example.yaml` document the allow-half migration. * fix(guardrails): address TypeSafe risk gate review findings - Status errors report only the numeric status. The HTTP status line is server-controlled like the body, and this message reaches middleware logs and the evaluation report, so a hostile endpoint could echo the state (raw tool arguments) through it. - Expired cache entries are removed with `pop(key, None)`. Sync evaluation runs on executor threads when a turn produces parallel tool calls, so a second caller can remove the same expired entry; a bare `del` raised KeyError, which the middleware reports as a provider error and, under fail_closed, turns into a spurious denial. - The evaluation harness states its acceptance gates inline (no missed dangerous call, system false-block <= 5%, uncached network p95 <= 1s) instead of citing a design spec that is not in the repository, and reports the whitelist-refusal population alongside the other four. - `--tools` / `--whitelist` strip whitespace and drop empty entries, so `--tools "bash, write_file"` no longer probes only bash while treating `" write_file"` as an unknown tool name. - The case set's `padding` note says top-level argument fields only, matching what the runner implements. Each fix was verified by running its new test against the pre-fix code. * refactor(guardrails): name the TypeSafe permission list allowed_tools Align the key with `AllowlistProvider.allowed_tools` so an allowlist migration copies it verbatim instead of translating `allowed_tools` into `whitelist`. The companion names follow: reason code `typesafe.tool_not_allowed`, metadata `tool_not_allowed`, harness population `not_allowed`, and the `--allowed-tools` flag. Semantics are unchanged: omitted = not enforced, `[]` = refuse every tool, tools outside the list are refused locally before any state, request or cache entry, and listed tools still face the risk gate. * fix(guardrails): record only a bounded served model token; fail the eval exit code Two findings from the re-review at d3f0b072. `model` is response content, and `_model_decision` interpolated it verbatim into the reason message -- which reaches the denied ToolMessage, the run journal, middleware logs and the evaluation report on disk, and which the cache retains for replay. A malformed or hostile endpoint can put an echoed tool argument, injected text or a multi-megabyte string there. The token is now recorded as itself only when it matches a conservative shape; anything else is recorded as `unrecorded:sha256:<digest>`, which keeps the verdict and a stable provenance record without echoing the value. Rejecting the response instead would turn an odd version string into a fail-closed denial of every call the endpoint answers. The digest path encodes with `surrogatepass` so an escaped lone surrogate -- a value `json.loads` happily produces -- cannot raise where this must not fail. `scripts/eval_typesafe_risk_gate.py` returned 0 even when a gate failed, so an operator scripting `... && enable` got a green light on a failing evaluation. `main()` now returns 1 when any gate in `report["score"]["gates"]` fails, after the JSON report is written, and prints the failed gate names. Tests: `test_an_unrecordable_served_model_is_digested_instead_of_echoed` (an echoed argument, a space, a lone surrogate and an oversized value, plus the cached replay) and `test_main_exit_code_carries_the_gate_verdict` (both directions). Both fail against the pre-fix source. docs/GUARDRAILS.md audit boundary updated to match. 113 passed: tests/test_typesafe_guardrail_provider.py, tests/test_typesafe_risk_eval_script.py, tests/blocking_io/test_typesafe_guardrail_provider.py. * fix(evals): fail the risk-gate run when a labeled population is uncovered Review finding on dcd38cc3. The new exit code made `_score`'s vacuous passes load-bearing: unprobed calls are excluded from the score, so `risky_misses_zero` is true over an empty risky population. `--tools read_file` excludes all 18 risky cases in the bundled set, so the run printed every gate as PASS and exited 0 after evaluating no risky operation at all. `_score` now gates coverage itself: `risky_coverage_present` and `safe_coverage_present` fail when the probe scope leaves that labeled population with nothing evaluated, so a scope that cannot substantiate the verdict no longer reports one. The safe side had the same vacuity -- `system_false_block_within_target` passes trivially with zero safe cases. Verified end to end against a loopback mock endpoint with the bundled case set: `--tools read_file` exits 1 with `GATE risky_coverage_present: FAIL` while the other four gates still PASS, where dcd38cc3 exits 0 on the same command. Tests: `test_main_fails_when_the_tool_scope_leaves_no_risky_case` and `test_main_fails_when_no_safe_case_is_in_scope` (both fail against dcd38cc3), `test_bundled_case_set_keeps_both_labels_the_coverage_gates_require`, and the exit-code test now runs with both populations present. 116 passed: tests/test_typesafe_guardrail_provider.py, tests/test_typesafe_risk_eval_script.py, tests/blocking_io/test_typesafe_guardrail_provider.py.	2026-09-24 18:20:31 +08:00
zeng-bohan	82ac1612e3	feat(storage): add a content-addressed blob store contract (issue #4189 item 2) (#5361) ... * feat(storage): add a content-addressed blob store contract (#4189 item 2) Addresses the review on the previous head: * the retention/dedup claim in the doc now states the deletion rule (liveness from the durable references) instead of promising refcount-free deletion, and the sidecar records the first writer as advisory provenance only; * one validate_blob_kind() owns the kind grammar, so the message a producer sees is the pattern that is enforced; * adds the missing storage/AGENTS.md (registry entry included) that both docstrings already pointed at; * drops the dead _import_backend_module() by routing the folder scan through it; * exports BlobNotConfiguredError from the package; * renames _KIND_MAX_BYTES to _MAX_BLOB_BYTES (it caps one put, not a kind); * delete failures raise the neutral BlobStoreError instead of BlobReadError. Rebased onto current main so the guidance-budget test matches upstream. Signed-off-by: zeng-bohan <zengbh1@gmail.com> * fix(storage): restore the config module and backend packages dropped by the squash The force-push to e72b435 silently lost four items the surviving code imports, breaking `import deerflow.storage` and the test suite at collection: - deerflow/config/blob_storage_config.py (BlobStorageConfig + get/set_blob_storage_config + load_blob_storage_config_from_dict, mirroring memory_config's host-shared schema and hot-reload pattern) - app_config.py wiring: the blob_storage AppConfig field, the import, and the _apply_singleton_configs -> load_blob_storage_config_from_dict call (memory stayed wired; blob did not) - storage/backends/__init__.py (drop-in contract docstring) - storage/backends/local_fs/__init__.py (LocalFsBlobStore re-export + STORE_CLASS for _scan_backends) Verified: pytest tests/test_blob_store.py tests/test_app_config_reload.py tests/test_app_config_name_indexes.py -> 67 passed; ruff format/check clean on all four files. * fix(storage): name the failing digest check; document the 64 MiB per-put cap - get_bytes: split the corruption message into size mismatch (truncation / partial write) vs digest mismatch (same-size bit rot, reporting what the content hashes to vs what the ref names) — the combined message reported 'got 1024 bytes, expected 1024' on same-size corruption and sent the debugger in the wrong direction. - docs/blob-storage.md + storage/AGENTS.md: state the _MAX_BLOB_BYTES cap (64 MiB) and that it is backend-level defense-in-depth rather than part of the BlobStore contract, so the externalized-tool-results follow-up sees the limit before the first production put fails. * test(storage): pin both get_bytes corruption branches separately The pre-split test corrupted the blob with different-size bytes, so only the size-mismatch branch ever ran and neither new message was asserted. Now: different-size corruption asserts 'size mismatch', and a same-size reversal (bit rot) asserts 'digest mismatch' — the branch the split was actually about. * docs(config): document blob_storage in config.example.yaml Add the blob_storage section (enabled/backend/backend_config) mirroring the memory section's shape, matching the Configuration section of backend/docs/blob-storage.md, and bump config_version 46 -> 47 per the example-file convention for schema additions. Default state is unchanged: enabled: false, purely opt-in. * fix(helm): align chart config version with example * fix(storage): refresh blob store on config changes --------- Signed-off-by: zeng-bohan <zengbh1@gmail.com> Co-authored-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: zeng-bohan <310902929+zeng-bohan@users.noreply.github.com>	2026-09-24 09:35:03 +08:00
RongJie G	e01314442c	fix(mcp): scope sessions and task access by thread incarnation (#5556) ... * fix(mcp): scope sessions and task access by thread incarnation * fix(mcp): preserve thread incarnation in delegated subagents * fix(mcp): preserve incarnation in durable batches * fix(mcp): bind standalone graph lifecycle context * fix(studio): preserve implicit thread creation metadata --------- Co-authored-by: CorgiBoyG <CorgiBoyG@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-23 08:33:20 +08:00
XuRuuuy	6102179c67	fix(browserless): read web_fetch wait timeouts like web_capture (#5702) ... web_fetch_tool parsed `wait_for_timeout_ms` with a bare int(), so any value that is not already an int raised ValueError and aborted the whole tool: a quoted number in YAML, or a typo like `2s`, turned every fetch into "Error: invalid literal for int() with base 10". web_capture_tool reads the same documented key through the tolerant `_as_int` helper two functions above, so the two tools disagreed about the same config key. While aligning them, web_fetch_tool now also reads `wait_for_selector_timeout_ms` instead of hardcoding 5000, so a `wait_for_selector` on a slow page can be given more than five seconds -- web_capture_tool already honours the key and the client has always accepted it. Co-authored-by: RXQ6 <RXQ6@users.noreply.github.com>	2026-09-22 19:47:46 +08:00
Zhengxu (Joshua) Jin	825e742eb6	docs(config): move the Sandbox banner above the sandbox key in config.example.yaml (#5652) ... * docs(config): move the Sandbox banner above the sandbox key in config.example.yaml The "Sandbox Configuration" banner and the "Option 1: Local Sandbox" comment sat directly above the `uploads:` block, twenty lines away from the `sandbox:` key they describe. Give `uploads:` its own banner and move the sandbox banner down to the key it introduces. Comments only; no key or value changes. * Enhance sandbox configuration options Updated sandbox configuration options to include BoxLite micro-VM. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-22 16:34:55 +08:00
JY-M666	1e6fda4ec2	docs: add llama.cpp local model configuration (#2931) (#5688)	2026-09-22 14:50:46 +08:00
spud	61a99a1e8a	fix(sandbox): enforce bash command timeouts in AIO sandbox (#5634) ... * fix(sandbox): enforce AIO shell command timeouts Map Sandbox.execute_command(timeout=T) onto the AIO legacy shell as a server-side hard_timeout=T plus a bounded host request (ceil(T+5)s, max_retries=0), preserve and render the upstream status (hard_timeout -> terminated + Exit Code: 124; no_change_timeout -> may still be running), keep no_change_timeout from preempting hard_timeout, and restrict ErrorObservation replay to completed results. * fix(sandbox): contain ambiguous AIO command outcomes Treat a transport timeout and statuses terminated/no_change_timeout/unknown as ambiguous: never replay the current command, fence the scoped or recovery session generation with bounded best-effort cleanup, and never adopt a replacement session for those statuses. hard_timeout and completed keep the session reusable. Align the env-bearing bash.exec path with the same contract and make its retry result authoritative. * fix(sandbox): wire AIO command timeout from provider config Restore bash_tool's non-local behaviour (other providers keep their own defaults) and let AioSandboxProvider read sandbox.bash_command_timeout as the sandbox's default_command_timeout at every construction site, with explicit per-call timeouts still winning. Widen the config field to float with allow_inf_nan=False and read the provider value with the module fallback so partial-config providers keep working. * docs(sandbox): document AIO command timeout semantics Document the provider-scoped deadline (LocalSandbox/AioSandbox/OpenSandbox wire it; other providers keep their defaults), the semver-vs-frozen-:latest image behaviour, the no-replay guarantees, and trim the sandbox guidance back under its size budget. * docs(sandbox): scope the wedge-resistance claim to command requests willem-bd's review on #5634 reproduced that list_dir and both create_session call sites still make unbounded SDK requests while holding the same sandbox lock, so the guide must not read as if every AIO request is bounded. State that the bounded T+5s host wait applies to command requests and name session-creation and file/list RPCs as not covered; bounding those remains a separate operation-deadline/control-plane follow-up. * fix(sandbox): make list_dir honor the shell generation fence list_dir previously always targeted the implicit persistent shell, so after an ambiguous command outcome fenced that generation (#5634) a listing could re-enter it. Route default-shell target selection through _ensure_default_shell_session_id so both execute_command and list_dir reuse the explicit recovery session, creating one only when the implicit shell is fenced. No list_dir deadline, hard_timeout, request budget, retry, or status semantics are added; those remain scoped to #5644.	2026-09-22 11:29:38 +08:00
xiaodu55	2ff006b0c0	feat(middleware): add deterministic PII redaction for model-bound context (#5527) ... * feat(middleware): add deterministic PII redaction for model-bound context * fix(middleware): claim national IDs before cards, redact Command results, preserve ToolMessage fields - Reorder detectors so checksum-gated national IDs run before the credit-card detector; an 18-digit resident ID whose digit run also passes Luhn is no longer mislabeled [CREDIT_CARD_n] (review finding, reproduced at 0a2a9d0) - Redact ToolMessages carried in Command.update.messages, mirroring ToolResultSanitizationMiddleware's dc_replace pattern - Rebuild redacted ToolMessages via model_copy so artifact and response_metadata survive - Extend the numbered middleware chain in agents/middlewares/AGENTS.md * fix(middleware): span one redactor per Command result; refresh stale AGENTS.md entry range - Placeholder numbering now continues across every ToolMessage carried in a single Command result (one _Redactor per _redact_result call) instead of restarting per message - The renumbered AGENTS.md chain still referenced entries 9-12 in the ToolReceiptMiddleware entry; it now reads entries 10-13 * docs(agents): trim PiiRedactionMiddleware entry to fit the AGENTS.md chain budget The main merge (fb36e0e) pushed the effective middlewares chain to 98341 bytes, 37 over the 98304 hard limit checked by agent-guidance (AG002). Compress the entry while keeping the load-bearing facts: config gate, both interception points incl. Command coverage, detector order rationale, per-result numbering continuity, irreversibility, memory follow-up. * fix(middleware): redact compaction input and reinjected summaries; harden detectors Review round 3 on #5527: - [P1] SummarizationMiddleware invokes its summary model directly from before_model, outside PiiRedactionMiddleware's wrap_model_call, so raw thread state reached the summary model and reinjected summaries carried raw PII into model-bound context. Add a shared redact_text() seam: the compaction prompt is redacted in _build_summary_prompt (app_config already flows into the middleware) and DurableContextMiddleware redacts summary_text at reinjection via a new pii_redaction_config knob wired at both assembly sites. - [P2] CUIT is 2+8+1 digits, not 2+10+1. - [P2] Digit-anchored patterns use digit-aware lookarounds instead of Unicode \b, which CJK characters defeat (身份证110105… / 手机号138…). - [P2] The international-phone pattern no longer treats newlines as separators, so a candidate cannot swallow the following numeric field and then fail validation as a whole. * fix(pii): redact title input and reserve summary placeholders --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-19 11:26:46 +08:00
zhangwei-way	b6503e9a35	feat(knowledge): add per-message RAGFlow retrieval scope (#5238) ... * feat(knowledge): integrate RAGFlow retrieval and management * test(knowledge): cover merged listing tool * feat(knowledge): add per-message retrieval scope * chore(docs): remove unrelated document * docs(knowledge): add interaction screenshots * feat(knowledge): simplify scope selector trigger * docs(knowledge): refresh selector screenshot * feat(knowledge): defer standalone management * docs(knowledge): show chat-only scope UI * fix(knowledge): honor scope on clarification replies * fix(knowledge): harden scoped replay validation * docs(knowledge): clarify replay scope precedence * fix(knowledge): keep provider settings on tools * fix(config): preserve tools-only knowledge settings * fix(knowledge): submit custom assistant identity * refactor(knowledge): trim PR scope changes * fix(knowledge): sanitize document scope display * feat(knowledge): enable scope selection in main chat * fix(knowledge): emphasize active scope icon without button frame * fix(knowledge): close context scrubbing and refresh e2e checks * fix(knowledge): preserve idempotent canonical retries * fix(knowledge): accept promptless conversation runs * style(knowledge): format backend regression tests * chore(knowledge): trim PR scope and fix frontend format * fix(knowledge): remove shared-scope notice * fix(knowledge): remove scope persistence notice * docs(knowledge): include main chat in catalog scope * fix(knowledge): preserve scope recovery and upgrades * fix(config): preserve LightRAG knowledge upgrades --------- Co-authored-by: foreleven <for-eleven@hotmail.com>	2026-09-18 16:59:31 +08:00
hataa	d811143b52	feat(authz): filter per-caller skill visibility on the skill listing surfaces (#4063 Phase 4) (#5489) ... * feat(authz): filter per-caller skill visibility on the listing surfaces (#4063 Phase 4) GET /api/skills, GET /api/skills/custom, and GET /api/skills/{name} now filter the user-scoped catalog through filter_resources(principal, "skill", ...) — mirroring list_models. Anonymous callers are unfiltered; provider errors follow authorization.fail_closed (fail-closed -> empty listing / 404, fail-open -> full listing). An invisible skill on the detail surface returns the standard 404 so the endpoint cannot become an existence oracle the filtered list closed. Management endpoints stay require_admin_user-gated; runtime activation is #4541's layer. resolve_skill_authorization joins resolve_model_authorization as a thin sibling over a shared _resolve_route_scoped_authorization core. * docs(authz): reflect per-caller skill visibility in OpenAPI metadata and implementation notes (#5489) Address the two non-blocking review findings on #5489: - The three user-facing GET routes (/skills, /skills/custom, /skills/{name}) now say in their /docs-visible descriptions that authorization filters the response (hidden skills 404 on detail). - Add the dated Phase 4 decision-log entry to the authorization implementation notes, per the convention of every prior merged authz PR: listing-visibility semantics, the 404-vs-403 existence-oracle rationale, anonymous-caller behavior, and the #4541 rebase reconciliation points (config.example.yaml roles comment + this file's decision log). * docs(authz): move route guidance into Gateway module guide --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-18 08:02:52 +08:00
Weng Qiang	a23dbdd837	feat(tavily): support configured search domain filters (#5513)	2026-09-18 07:25:49 +08:00
Hyeonsang Cho	53798b44cd	fix(subagents): scale max_turns into the graph's super-step budget (#5485) ... * fix(subagents): scale max_turns into the graph's super-step budget max_turns was handed to LangGraph as recursion_limit, but the two count different things. recursion_limit counts super-steps, one per graph node, and create_agent compiles a node for every middleware lifecycle hook, so one turn costs before_model + model + after_model + tools nodes — seven to eight through the subagent chain. The built-in general-purpose agent's max_turns=150 therefore bought about 18 tool-using turns before failing as turn_capped, and every middleware added to the chain shrank the effective budget again. Resolve the limit from the chain each subagent was actually assembled with (subagents/turn_budget.py) instead of passing the turn count through, so raising max_turns buys the turns it names. No config keys or defaults changed; existing max_turns values now grant their full budget, bounded as before by subagents.timeout_seconds and subagents.token_budget. * fix(subagents): warn when a counted hook can jump the agent loop Review follow-up. The per-turn cost is a flat multiplier over the straight before_model -> model -> tools loop. A hook that declares can_jump_to and returns {"jump_to": ...} re-enters the loop without traversing tools, spending another before_model + model + after_model pass that buys no tool result, so the resolved limit becomes a lower bound rather than an exact budget — silently re-creating the short budget this translation fixes. Measured against a compiled graph: with one jumping after_model hook, three tool turns need the resolved limit plus one jump pass, and the run raises GraphRecursionError at the resolved limit. How often a jump fires is data-dependent and unbounded, so it cannot be folded into the arithmetic. find_jumping_hooks reports the condition off the same __can_jump_to__ attribute the factory reads, and the executor warns when a counted hook declares one. Nothing in today's subagent chain does, so this changes no budget. * fix(subagents): detect jumps declared on agent-level hooks too Review follow-up. find_jumping_hooks exempted before_agent/after_agent on the grounds that a jump out of them lands in the loop the budget already pays for. That does not hold on langchain 1.3.14: - after_agent jumps re-enter the loop after it finished, and the hook runs again on the next exit, so the extra passes are unbounded. Even jump_to "end" is routed to exit_node, the head of the after_agent chain, so it reruns the chain; destinations are no safe filter. - a before_agent hook that stages a tool call and jumps to tools runs a tools step no model turn paid for. It is O(1), but the resolved limit has zero headroom, so one step caps the last turn. The detector now scans every hook pair the factory wires jump edges for. The compiled-graph pin is parametrized over after_model->model, after_agent->model, after_agent->end and before_agent->tools, each raising GraphRecursionError at the resolved limit and completing once the jump's cost is added. No middleware in the subagent chain declares a jump on any hook, so this changes no budget.	2026-09-17 09:05:25 +08:00
Onefly	d8d110c637	fix(sandbox): prevent AIO subagent session eviction (#5178) ... * fix(sandbox): prevent AIO subagent session eviction * fix(sandbox): address PR 5178 review issues * fix(sandbox): handle transient session and metadata failures * fix(sandbox): fence capacity upgrades and validate reused limits * docs(sandbox): restore list indentation and trim guidance * fix(ci): stabilize Buzz persistence test and trim sandbox guidance --------- Co-authored-by: ranxi2001 <ranxi2001@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-17 07:45:19 +08:00
Totoro	0efdf8e7d8	fix(sandbox): cut read_file output at a line boundary and name the next start_line (#5474) ... * fix(sandbox): cut read_file output at a line boundary and name the next start_line read_file head-truncates at a character offset and its marker told the model to continue with start_line/end_line while reporting only character counts, so the cut usually fell mid-line and the model had to guess which line to continue from. The cut now lands on the last line boundary the budget allows, and the marker reports lines shown of lines total, keeps the character counts, and names the exact next start_line. When the line at the cut is longer than 4,096 characters (minified sources, one-line JSON) the cut stays at the character limit and the marker names the line it fell inside, so a re-read of that line is the continuation. Reads under the limit are unchanged. * fix(sandbox): make the read_file continuation hold for ranged reads and long lines Line numbers in the truncation marker are now file line numbers: read_file_tool passes start_line - 1 as the line offset, so a ranged read that is itself truncated names the right next line instead of one relative to its slice. A ranged read is a provider slice joined with newlines, so the tool also says so and a trailing newline there counts as an empty last line. The long-line fallback now names a continuation only when it makes progress: a read from the cut line when the whole line fits such a read, a single-line read (start_line = end_line) when only the line alone fits max_chars, and bash when even that cannot return it; the single-line form names no further line after the last line of the read. The budget reserves one extra character so a newline sitting exactly at the limit still counts as a complete line, the "fits a fresh read" check uses a pessimistic estimate of the follow-up read's marker, and a budget too small for any marker still returns a marker instead of a bare prefix. Adds unit cases for the ranged-read offset, the newline-at-budget edge, the single-line-read and bash forms, tiny budgets and empty last lines, plus end-to-end tests that drive read_file_tool with a LocalSandbox and follow the markers across reads, asserting the kept segments reproduce the file without gap or overlap. * fix(sandbox): keep naming the next line after a bounded read's last line A ranged read with an end_line below the file's length is a slice that stops mid-file, so the single-line-read continuation must still name the line after the slice's last line; only a read that reached the end of the file names nothing further. The tool passes whether the read was bounded by an end_line separately from the joined-lines hint, because a start_line-only read also runs to the end of the file. A blank line and a line past the end both read back as an empty slice; the tool now tells them apart with a two-line probe, so a continuation named by a marker that lands on a blank line answers "(empty)" rather than "(start_line exceeds file length)". * test(sandbox): adapt upstream continuation checks after rebase --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-16 23:57:38 +08:00
Zeren Wang	a58ab484a6	feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash (#5443) ... * feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md (issue #5160, tracker #5129) in the slice order of the spec's §16. Slices: - A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY admission pinning (both server-owned sets + worker hoist); latest-only request-scoped <project> block via DynamicContextMiddleware wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID prefix + marker + provenance, never persisted); journal audit fingerprints; Instructions tab. - B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with locked check-and-set; hash-qualified immutable shelf storage with Paths helpers; upload/list/content/delete-to-trash routes; project delete trashes the shelf in-transaction; request-scoped bounded <documents> index with honest count/shown + actionable overflow note; list_project_documents/read_project_document tools registered only on pinned runs; PAT allowlist + drift guards; blocking-IO anchors. - C: shared thread-upload ingestion service (uploads router refactored to parity); POST from-thread with provenance; attach-to-thread with lock-staged copy (archived source allowed); read-only thread-files view with per-group truncation reporting. - D: restore (restored/merged/not_found/no_target/content_missing; no file moves), purge (continuous row lock across unlink/delete/commit, retryable on FS errors), retention sweep (lazy + startup, 24h orphan guard, row-side reconciliation never deletes). - E: Documents tab (shelf + conversation-files browser, provenance, archived banner, content-missing rows), /workspace/trash route, sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks + specs. Review hardening folded in (10 rounds, all with tests): - force active shelf content (HTML/XML family) to download; nosniff on artifact + content responses; unified unsandboxed-iframe PDF preview (fixes the pre-existing Chromium sandbox blank in the artifact viewer) - scope document trash to the URL project under the document lock - atomic no-overwrite filename reservation for ALL ingestion (seeded claims + os.link commit with suffix retry; same-name re-upload now unique-names instead of replacing); hidden staging only, no visible placeholders; lease cleanup on setup failure - serialize conversion under the document lock with post-lock active revalidation; drain locked filesystem work on cancellation; preserve bytes when an insert's commit state is uncertain (including trashed rows) - original-integrity checks before serving text or cached conversions; content_missing surfaced in list responses (UI reads the flag, no 409-probe); downloads always serve original bytes - bounded streaming document reads with cached char counts; shelf limits declared in middleware release identity - thread-root confinement for from-thread sources; config fallback rejects fractional/infinite values; composer counts staged attachments; pending attachments persist until submission or removal; in-flight instruction/rename edits survive save refetches; shelf and trash pagination; conversation-file and thread-files pages stay subscribed to refetches Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md contracts, config.example.yaml projects block. Review follow-ups (head b4807477 → this revision): - The trash retention sweep is split so repeated lazy triggers stay bounded: the indexed expiry purge still runs on every trigger (GET /api/trash/documents, POST /api/trash/purge) while the O(all rows + all files) reconciliation is throttled to one run per user per 15 minutes (process-local, per-user window). The startup sweep now runs as a background task instead of blocking gateway readiness, and shutdown awaits it (bounded). - The export scrub (stripInternalMarkers) is fence- and indentation-aware like the render path, so a pasted, fenced <project>/<documents> snippet survives markdown export while real injected blocks (never fenced) are still removed. Fence regexes moved to a dependency-free leaf module to avoid the messages↔streamdown import cycle. - The artifact viewer's PDF iframe no longer carries an added title attribute (the upstream e2e contract locates it via :not([title])), and the upstream artifact-preview spec now pins the new contract: PDFs render unsandboxed, images keep sandbox="". * fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title - Shutdown cancelled only the shield around the background startup sweep, so an all-users reconciliation that outlived the 5s budget kept walking rows and files while the document repo and DB engine were disposed underneath it. The wait now lives in `_shutdown_startup_trash_sweep`, which cancels the task and drains it before worker exit: the shield keeps the wait bounded, the cancel makes it final (CancelledError lands at the sweep's next await, and `_run_startup_trash_sweep` only catches `Exception`, so nothing swallows it). - The browser-preview iframe lost `title={getFileName(filepath)}` in the previous fix round, leaving the PDF frame without an accessible name while its siblings keep theirs. Restore it (WCAG frame titles), assert it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]` instead of `iframe:not([title])`. * fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel `Task.cancel()` returns False when the sweep already finished inside the window between the deadline firing and the cancel, so the shutdown log claimed a cancellation that never happened. Branch on that outcome: the warning stays for a real cancel, a late finish is logged at info, and both paths still reap the task before worker exit. * fix(projects): round-4 review — make Empty trash delete what it confirms `POST /api/trash/purge` only ran the retention sweep, and the sweep's candidate selection is age-gated, so a freshly trashed document survived "Empty trash" even though the confirmation promises that every listed document is permanently deleted. With one trashed row the route answered `{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps expired rows before listing, so the visible rows were normally ineligible for the action by construction. Empty trash now drives `purge_all_trashed`: the caller's trashed rows (`list_all_trashed`, no age filter) each go through the same guarded, row-locked `purge` as the single-document delete — bytes first, then the row, in one transaction — so a row restored mid-flight is skipped instead of force-deleted, and an unlink failure rolls that row back and answers 500 with a retryable message. Retention expiry stays where it was: the sweep's `purge_candidates` is now the only age-gated selection, and the lazy retention sweep still runs on the listing and at startup. Tests: the router suite replaces the retention-gated expectation with the reviewer's repro (fresh row purged, bytes unlinked, shelf and other users' trash untouched, a failing unlink stays retryable and 500); a blocking-I/O anchor drives the new entry point through the offload; the mocked e2e covers the action end to end; a new real-backend spec performs it against the real gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE and the phase-2 design docs (en+zh) state the age-independent contract.	2026-09-16 18:46:18 +08:00
hataa	26800d1245	fix(channels): stream-cap and validate WeChat/WeCom inbound media downloads, fixes #5223 (#5225) ... * fix(channels): stream-cap and validate WeChat/WeCom inbound media downloads, fixes #5223 * fix(channels): address WeCom APPID, decompression, and log-sanitization review findings (#5223) Round-4 review follow-ups on the inbound-media download cap: - The COS bucket numeric suffix is the owner's Tencent Cloud APPID and bucket names are user-chosen, so any Tencent Cloud account could register a matching ww-aibot-img-* bucket and pass the shape gate. The built-in rule now admits only the APPID observed in Tencent's published aibot callback examples (1258476243), across regions; any other account (including a future WeCom rotation) goes through channels.wecom.allowed_media_hosts. - aiter_bytes() transparently decodes Content-Encoding, and the decoder allocates the full decompressed body before the byte cap sees a chunk (an ~8 KB gzip wire chunk decoding to 8 MiB reproduces it). Both URL readers now send Accept-Encoding: identity, refuse a response with a residual Content-Encoding before reading, and iterate aiter_raw(). - httpx.HTTPStatusError formats the signed URL (path + query credentials) into its message, so _ingest_inbound_files' reader-failure branch logs a sanitized summary (class + status) instead of logger.exception, and the WeChat extract paths catch httpx.HTTPError so the polling loop's per-message logger.exception can never render a media URL. Every change ships with a red/green regression: the reviewer's 403 mock-transport repro asserted against caplog.text (fully formatted logs), the reviewer's different-APPID bucket host, and gzip bombs driven through real httpx mock transports in both readers. Docs (channels AGENTS.md, README, config.example.yaml) updated for the APPID pinning and encoding gate. * docs(channels): document why the inbound-media cap is 50 MB, not WeCom's 100 MB ceiling * fix(logging): redact URLs in httpx request logs down to scheme + host, fixes #5223 httpx emits 'HTTP Request: GET <full URL>' at the Gateway's INFO level before any response handling runs, so even successful signed-media downloads leaked their credentials. HttpxUrlQueryRedactionFilter (installed by configure_logging) rewrites those records in place — path and query become /<redacted>, method/status/duration observability is preserved — which also keeps Telegram's token-bearing Bot API paths out of the logs. Reader-level regression tests run at production INFO level with a real MockTransport, success paths included. * fix(logging): blank userinfo credentials in httpx request-log redaction * fix(logging): redact authority-only URLs and cover urllib3 redirect logs Two follow-ups from the review plus one extrapolation of the same class: - rest is now optional in _URL_REDACT_RE, so an authority-only URL (scheme://user:pass@host, no path) is rewritten too — userinfo had nowhere else to hide and previously passed through verbatim. A bare credential-free origin still passes through unchanged. - Renamed to UrlRedactionFilter / install_url_log_redaction and attached to the urllib3 logger as well: urllib3 logs 'Redirecting <url> -> <url>' at INFO with full URLs on both sides, the same leak class on a different library logger. No gateway path today both uses requests and redirects a signed URL, but the class stays closed instead of dormant. - Unit tests now build records with the real httpx 0.28.1 format string ('HTTP Request: %s %s "%s %d %s"', 5 args) and httpx.URL args, per the nit, instead of a synthetic shape httpx never emits. * fix(logging): install URL redaction at handler level so propagated records are covered A logging.Filter on a logger only runs for records emitted through that exact logger — child loggers neither inherit it nor trigger it on propagation — so the previous attachment to the bare urllib3 logger was dead code: urllib3 emits Redirecting via urllib3.poolmanager at INFO and urllib3.connectionpool at DEBUG. The filter is now attached to every root handler (mirroring _install_trace_filter, which already iterates root handlers; handler-level filters see propagated records) in addition to the httpx logger (httpx emits via the bare name, and emission-point coverage survives handlers added later). The wiring is pinned by tests that emit through the real urllib3 child loggers — a mutation removing the handler-level install turns them red. Comments, docstrings, and AGENTS.md now state the actual emitter names and levels. * fix(logging): redact urllib3 DEBUG request lines, whose split shape evaded the URL regex urllib3's per-request line (connectionpool.py:545 on 2.7.0) renders as `scheme://host:port "METHOD /path?query HTTP/x.x" status len` — the authority ends at a space so _URL_REDACT_RE's bare-origin early return applies, and the quoted origin-form target has no scheme, so neither half was rewritten. UrlRedactionFilter now runs a dedicated request-line shape first (collapsing the target to /<redacted>, keeping scheme+host+method+ version), then the absolute-URL pass. Regressions pin the exact format string both at unit level and through the real urllib3.connectionpool DEBUG emit path; AGENTS.md wording now names both covered DEBUG shapes. * fix(logging): redact urllib3 retry lines and linearize scheme scanning Closes the two open review threads on the inbound-media log hardening: Retry/redirect targets: urllib3 logs the request target with no scheme in five shapes the generic absolute-URL pass cannot see - `Retry: <target>` (connectionpool.py:954 DEBUG), `Incremented Retry for (url='<target>')` (util/retry.py:545 DEBUG, absolute on the redirect path), `Retrying (...) after connection broken by '<err>': <target>` (connectionpool.py:869 WARNING, above the INFO root), and origin-form halves of both Redirecting emitters (poolmanager.py:500 INFO / connectionpool.py:922 DEBUG). Each gets a rewrite anchored to the exact urllib3 format, collapsing the target to /<redacted>; the generic pass's rest now stops at quote characters so a quoted URL keeps its closing punctuation (previously the absolute-form increment line was mangled), and the request-line method class accepts any case. The emitter enumeration in channels AGENTS.md is closed against the installed urllib3 2.7.0 source. Quadratic scanning: both scheme-bearing patterns start with a character class, so re.sub retried every suffix of a long token - 64K paths cost ~1.8s and URL-free 64K error bodies ~3.1s per record, synchronously in every root handler. The two passes are now driven from literal "://" occurrences: _scheme_starts walks back over the scheme charset to each run's first letter and the pattern is attempted only there, reproducing re.sub's leftmost-non-overlapping result in linear time (256K path: 5.6ms; worst adversarial shapes <= 28ms). Long-input regressions pin the URL-bearing and URL-free cases with mutation-verified bounds, plus nested-scheme and digit-headed-run equivalence cases. Validation: tests/test_logging_config.py 12/12; scheme-pass equivalence against the old re.sub pipeline verified by two independent 30k+ case fuzz runs; full-suite A/B against HEAD shows zero tests that pass on HEAD and fail with this diff. * fix(logging): boundary-aware quote stops and whole-message Redirecting anchor Two follow-ups on the urllib3 redaction shapes: Embedded quotes: `rest` treated ANY quote as a closing mark, so a URL with an apostrophe in the path kept everything after it verbatim (`https://h/path'quoted'?token=Q` rendered the credential suffix in full) while the class docstring claimed path/query/fragment are replaced. A quote now closes `rest` only at a boundary - followed by whitespace, a closing parenthesis, or end of string - so urllib3's Incremented Retry (url='...') scaffolding keeps its ') closer while an embedded quote stays consumed. The increment line's url capture gets the same rule narrowed to its fixed ')' closer. Redirecting anchoring: the origin-half pass matched `(? <=-> )/path` as a substring, and an `-> /path` arrow is not urllib3-owned shape - the sandbox provider's actionable mount error (`sandbox.mounts entry <host> -> /mnt/knowledge ignored: ...`) had its container path rewritten to /<redacted>, failing test_setup_path_mappings_logs_actionable_error_for_missing_host_path on CI (backend-unit-tests shard 3). The pass is now anchored to the whole `Redirecting <t> -> <t>` message, which is exactly urllib3's record; origin slots collapse, absolute slots stay for the generic pass. Regression tests pin the embedded-quote shapes and the sandbox error's byte-for-byte passthrough; both mutations verified red. Validation: tests/test_logging_config.py 14/14; the CI-failing sandbox test green locally; every test file asserting redaction/arrow log content passes (attachments, support bundle, run metadata, skill secrets, ragflow, skillscan, sandbox provider); full offline backend suite 14084 passed / 164 failed with the failure set matching this machine's documented Windows-environment baseline (NTFS chmod/symlink, docker/lark/langfuse absences) - no failure involves redaction output. * fix(logging): redact redirects with spaced locations * fix(logging): grammar-complete Redirecting anchor; neutral WeChat guard labels Round-13 P3 (Redirecting anchor strictness): the whole-message anchor kept the ^Redirecting prefix (the urllib3-owned literal that stops the sandbox false positive) but required BOTH slots whitespace-free, so a Location header with an interior space voided the pass and leaked the origin-form request target in the first slot - redirect_location is the raw header string and interior spaces are legal field syntax. The tail is now loose (\S.*$) and the first slot gets the same grammar treatment (\S.*?): the recursive urlopen frame passes the previous raw Location as its url, so t1 can carry interior spaces too, lazy-split at the first arrow the way the line is constructed. A space-carrying slot collapses whole when it starts with /; the sandbox mount error keeps passing through untouched. Round-14 nit (None conflation): _download_cdn_bytes returns None for two reasons (in-flight cap abort, Content-Encoding refusal) but both image and file callers labeled it "exceeds size limit (N bytes)" - contradicting the accurate encoding line right above it, and reporting the plaintext limit for a ciphertext-cap decision. Callers now log a neutral "skipped by download guard" line (the manager reader callers' shape); the accurate reason stays inside the download function. The same sweep also logs _stage_downloaded_file's silent None (no state dir configured), which made an attachment vanish with no log line at all. Also anchors the emitter-enumeration closure to its urllib3 version: the closure reopens if an upgrade changes these format strings, so the comment now says so explicitly. Validation: logging 15/15 and attachments 60/62 (the two pre-existing Windows symlink-privilege failures documented in the PR body); three mutations verified red (old wording, strict t1, silent staging None); ruff clean. Full offline suite run before push (per round-11 lesson). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-16 16:33:47 +08:00
yang rui	3dc895df4d	feat(models): pace shared RPM budgets before dispatch (#5432) ... * feat(models): add shared RPM admission queues * fix(models): address admission pacing review feedback * docs: simplify request admission quick start guidance --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-15 07:14:35 +08:00
Totoro	533e30e7f2	[feat] add opt-in conversation reads for Gateway runs (#5399) ... * feat: add scoped conversation reads to gateway runs * refactor: share conversation tool path and reuse parsed text --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>	2026-09-14 10:51:01 +08:00
cybersentia	a22c6169b3	feat(skills): support OpenAI-compatible image generation (#5389) ... * feat(skills): support OpenAI-compatible image generation * fix(skills): address image provider review feedback	2026-09-13 18:07:58 +08:00
Nan Gao	cf556fa9d4	feat(agents): elide superseded write_file payloads from model-bound requests (#5374) ... * feat(agents): elide superseded write_file payloads from model-bound requests Step 2 of #5328. After a successful write_file the file on disk is the source of truth, and the read-before-write gate forces a read_file before the next modification of that path, so once a later successful read or write of the same path exists the historical `content` argument is redundant with it. Long report-writing runs (append-in-chunks) therefore carried every section twice, once as the write argument and once as the following read output, until summarization compacted the whole turn. - ToolOutputBudgetMiddleware's model-call hooks now replace such superseded content with a short deterministic placeholder pointing at read_file, in the model-bound request only: state["messages"], checkpoints, receipts, loop detection, and the run journal keep the original arguments, and nothing is externalized to disk. The newest `keep_recent_writes` successful writes (default 1) always stay visible; str_replace payloads are never touched; a same-turn read never supersedes (parallel calls run in no fixed order); only results stamped deerflow_tool_meta.status == "success" count, so failed, gate-blocked, partial, or unstamped writes are never candidates. - New `tool_output.elide_superseded_writes` (default on), `tool_output.superseded_write_min_chars` (default 2000), and `tool_output.keep_recent_writes` (default 1); config_version 41 -> 42 in config.example.yaml and the Helm chart. - The per-occurrence call/result pairing the gate introduced in #5329 moves into the shared `tool_call_args.pair_tool_call_results` helper so both policies pair the same way; the gate now uses it. * fix(agents): scope tool-call result pairing to the issuing turn Review finding on #5374 (P2): pair_tool_call_results consumed results from a history-wide per-id queue, so an interrupted write_file with no result whose tool-call id a later turn reused inherited that later call's success. With the default elision the unconfirmed draft was then replaced by a placeholder claiming the write succeeded, and the gate's blocked-call pairing had the mirror-image hole. Pair results the way DanglingToolCallMiddleware does: walk in document order, open each AIMessage's calls, and let a ToolMessage answer only a still-open call of the most recent preceding AIMessage. A result never answers a call from an earlier turn, so the interrupted call stays unanswered (never a candidate, never labeled blocked) and stray or duplicate results are ignored. Regressions cover the helper, the superseded-write policy, and the gate. * fix(agents): never rewrite tool-call ids duplicated within one AIMessage Review finding on #5374 (P2): the policies select calls per occurrence, but every provider surface is addressed by tool-call id, so when a malformed provider payload repeats an id inside one assistant turn the rewriter could only replace all of its occurrences at once. A failed write_file sibling then took on the superseded successful call's path and elided content and was presented as a success; the gate's blocked-call elision had the mirror-image hole (a successful sibling rewritten into the blocked call). rewrite_messages_tool_call_args now never offers an id that repeats within its message to the selector and leaves those calls untouched on every surface. Both policies are covered by the shared helper; regressions cover the helper, the superseded-write policy, and the gate. * fix(agents): skip unhashable tool-call ids in the duplicate-id guard Review finding on #5374 (round 3): _duplicated_call_ids fed every id into a Counter before the string guard, so a list or dict id from a malformed provider payload raised TypeError out of wrap_model_call and failed the whole model call whenever the history also held a rewrite candidate. The pre-PR loop and pair_tool_call_results skip such ids; only this helper regressed. Count non-empty string ids only, and pin it with regressions for the helper, the superseded-write policy, and the gate. * docs(agents): keep middleware guidance within size limit --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-13 18:04:51 +08:00
Aleksandr Sapronov	b448c02bbe	fix(firecrawl): pass tool base_url to FirecrawlApp as api_url (#5392) ... _get_firecrawl_client only read api_key from the tool config and ignored base_url, so self-hosted Firecrawl deployments were unreachable: the SDK defaulted to https://api.firecrawl.dev and raised 'No API key provided'. Now base_url is forwarded as api_url; api_url is omitted when unset so cloud behavior is unchanged. Also document the optional self-hosted base_url on both Firecrawl entries in config.example.yaml and reconcile their headers to the fastCRW house style (Cloud requires FIRECRAWL_API_KEY; self-host may need no key.).	2026-09-13 17:23:04 +08:00
Xuehao Xu	56540fab01	fix(gateway): make recursion limit configurable (#5390) ... * fix(gateway): make recursion limit configurable * docs: keep backend guidance within inherited size budget * fix(gateway): address recursion limit review feedback * docs(gateway): clarify recursion default scope --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-13 17:08:33 +08:00
Wenchao An	1b76ab9060	feat: add opt-in task notes and compacted history recall (#5382) ... * feat: add opt-in task notes and compacted history recall * fix: validate task continuity state and preserve user answers Honor explicit opt-out, preserve clarification replies and capture failure statuses, validate notebook writes, and clear branch archive references. Update the config version and audit optional LLM credentials, with regression and integration evidence. * fix: align Helm config version with task continuity schema * fix: preserve mixed task history and declare continuity policies * fix: recover malformed history and evict archives atomically	2026-09-12 21:01:46 +08:00
wutongyuonce	4470932118	feat(extensions): allow constructor kwargs on config-declared middlewares (#5312) ... * feat(extensions): allow constructor kwargs on config-declared middlewares extensions.middlewares entries may be a class-path string or {class, kwargs}. String entries keep the zero-argument constructor. Unknown fields and blank class paths fail at config validation. Constructor errors still fail at agent creation. Fixes #5311 * fix(extensions): coerce middleware kwargs to JSON types YAML timestamps became datetime objects while JSON kept strings, so constructors and to_file_dict() json.dump saw different types. Validate kwargs as JSON types at config load, stringify dates, reject NaN and other non-JSON values, and cover the raw-dict loader branch. * style(extensions): wrap middleware Field description for ruff E501 make lint failed: the middlewares description was 289 chars (limit 240). Wrap it and run ruff format on the two files this PR last touched. * docs: compact configured middleware guidance to satisfy size limit --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-12 10:57:40 +08:00
Nan Gao	3f0b6ecc81	feat(agents): elide blocked write payloads from model-bound requests (#5329) ... * feat(agents): elide blocked write payloads from model-bound requests A write_file / str_replace call rejected by the read-before-write gate never runs, yet its payload (up to 80 KB for a non-append write, unbounded for append) stayed verbatim in every later model request: nothing in the chain rewrites AIMessage tool-call arguments, and ToolOutputBudgetMiddleware only budgets ToolMessage output. The gate demands a re-read plus a fresh call, so the model re-emits the content anyway and the original is pure dead weight. - ReadBeforeWriteMiddleware stamps `deerflow_write_block` on the blocked ToolMessage and, in wrap_model_call, replaces the paired call's payload fields (content / old_str / new_str) with a short deterministic placeholder in the model-bound request only. state["messages"], receipts, loop detection, and the run journal keep the original arguments; nothing is externalized to disk, since a file reference to content the model must re-derive after reading the target would only invite bypassing the gate. - New `tool_call_args` helper rewrites every provider surface together (structured tool_calls, raw additional_kwargs.tool_calls, tool_use content blocks, tool_call_chunks) so strict providers never see them disagree; the gate only supplies the policy (which calls, what placeholder). - `read_before_write.elide_blocked_payloads` (default on) and `read_before_write.elide_min_chars` (default 2000) configure it; the runtime builder passes the config through and the middleware declares it via release_policy_parameters. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(agents): condense middleware guide entry 11 to fit the guidance budget The agent-guidance CI check failed: the effective AGENTS.md chain for agents/middlewares was 99673 bytes against a 98304-byte hard limit. The chain already sat at 98459 on main, so the ReadBeforeWrite entry could not grow. Rewrite entry 11 so it states the same facts (gate, lock scope, fail-open, authorization scope, blocked-payload elision, shared tool_call_args helper) in 1229 bytes instead of 2640; the chain is now 98262 bytes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(config): bump config_version for the read_before_write elision keys Review follow-ups on #5329: - `read_before_write.elide_blocked_payloads` / `elide_min_chars` are new user-settable YAML keys, i.e. a config schema change, so bump `config_version` 40 -> 41 in config.example.yaml; without it an existing config.yaml gets no outdated-config warning and `make config-upgrade` has nothing to signal. - Say in the `elide_min_chars` description (and the example comment) that the threshold and the placeholder's size figure are Python character counts, not tokens: the same value spans roughly 3-4x in real context cost between ASCII and CJK text. - The builder wiring test now asserts only the wired `elide_min_chars` value instead of the whole `ReadBeforeWriteConfig` dump, so future knobs do not have to edit an unrelated test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(helm): bump chart config_version to 41 validate-chart's config_version drift check failed after config.example.yaml moved to 41 in ef9ee267. Bare bump of the chart's embedded `config:` block and the README example; the chart does not mirror the read_before_write section, so no field changes are needed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(agents): rewrite Responses and v1 content-block arguments too Review finding on #5329 (P2): the content rewriter only handled Anthropic `tool_use` blocks. With `use_responses_api=true` and `output_version='responses/v1'`, AIMessage.content carries `function_call` blocks whose `arguments` still hold the full write payload, and langchain_openai's Responses input builder emits that block instead of the rewritten structured call whose `call_id` it already carries. Standard `v1` `tool_call` blocks likewise keep `extras.arguments`, which the v1->Responses translator prefers over the structured args. So the blocked payload was still sent on every later Responses API request. `tool_call_args` now rewrites every content dialect that carries its own copy of the arguments: Anthropic `tool_use` (input, drop partial_json), Responses `function_call` (arguments, matched by call_id, `fc_...` item id and status preserved), and v1 `tool_call` / `tool_call_chunk` (args plus `extras.arguments`). Tests assert against the real adapter serializers: `_construct_responses_api_input` for responses/v1, v1, and v0 messages, `_convert_message_to_dict` for chat completions, and Anthropic `_format_messages` for native and v1 content, plus an end-to-end probe through the gate's wrap_model_call. * fix(agents): pair blocked writes per call occurrence and defeat Responses chaining Two review findings on #5329: - Tool-call ids may repeat across assistant turns (DanglingToolCallMiddleware pairs them with per-id queues). The gate matched blocked results against a history-wide id set, so a successful write sharing an id with a later (or earlier) blocked one also lost its payload and was labelled as blocked. `_blocked_call_occurrences` now pairs ToolMessages with call occurrences the same FIFO-per-id way and the selector keys on (message, call id). - With `use_previous_response_id`, the OpenAI adapter sends only the messages after the last AIMessage carrying a `resp_` response id and lets the server rebuild the rest from its stored copy, which still holds the original arguments and cannot be edited; every later response chains back to it. `rewrite_messages_tool_call_args` now drops every `resp_` id from the model-bound copy whenever it rewrote anything, so the adapter replays the full rewritten history (the `use_previous_response_id=False` request shape). OpenAI bills chained input tokens as input either way, so replay costs no more; the state keeps its ids. Tests cover success-before-block and block-before-success histories through the Chat Completions serializer, and chaining through `ChatOpenAI._get_request_payload` with `use_previous_response_id=True`: unrewritten history chains and omits the call, rewritten history is replayed with the placeholder and no `previous_response_id`. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-11 19:05:48 +08:00
Nan Gao	0cf1762638	docs(config): note use_previous_response_id in the Responses API example (#5359) ... `use_previous_response_id` is not a DeerFlow config field; it reaches `ChatOpenAI` only because `ModelConfig` allows extra keys and the model factory forwards them. It is easy to mistake for `use_responses_api`, but the two are not equivalent: the former picks the endpoint, the latter switches the Responses API from replaying the full history each turn to chaining on `previous_response_id` with server-side state. Chained context is still billed as input tokens, and client-side history rewrites are only honored when the history is replayed. Document the key next to the Responses example so operators know what it changes. Comment only, no schema change.	2026-09-11 16:22:26 +08:00
PeaceMaker-best	36ce7590b7	fix(agents): isolate loop detection state by run (#5344) ... * fix(agents): scope loop detection state per run * fix(agents): harden loop scope fallback * docs: move loop lifecycle detail out of inherited guidance --------- Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-11 13:45:52 +08:00
wutongyuonce	48a8978b7b	feat(scheduler): add interval schedule type (#5291) ... * feat(scheduler): add interval schedule type Allow scheduled tasks to fire every N seconds from last dispatch, not only wall-clock cron or a single run_at. Cadence is UTC now+N with no missed-beat catch-up, bounded by min_once_delay_seconds and 30 days. * fix(scheduler): let interval tasks create, edit, and keep next run Create/edit now keep every_seconds. Unchanged interval spec no longer resets next_run_at, including timezone-only PATCH. * fix(scheduler): keep non-minute intervals on edit Stop rounding every_seconds to whole minutes in the form. Values that are not whole minutes or hours now use a seconds unit so edit/duplicate round-trips the stored cadence instead of rewriting it and resetting next_run_at. Document that min_once_delay_seconds is also the interval floor. * fix(scheduler): clamp interval seconds to the default 60s floor The new seconds unit allowed 1–59, which the API rejects under the default min_once_delay_seconds. Clamp the form to >= 60 and show the floor next to the preview. Also mention interval in the scheduler field_doc, matching config.example.yaml. * fix(scheduler): do not clamp interval amount while typing Keystroke clamp made 90 become 9 -> 60, then 600, and backspace could not leave 60. Keep the raw field text and apply the 60s floor on blur and emit only. * test(scheduler): cover interval input editing * fix(frontend): preserve saved interval cadence until edited * style(tests): format scheduled task router tests --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-10 16:46:01 +08:00
Yusuf Gürdoğan	23bd76046a	feat(community): add Sofya web search provider (#5239) ... * feat(community): add Sofya web search provider Add a community provider backed by Sofya (https://sofya.co). Its search endpoint returns the content of the result pages, not only their snippets, and its fetch endpoint returns a page as markdown. Both are plain JSON over HTTP, so this needs no extra Python package (uses httpx, already a dependency). Changes: - backend/packages/harness/deerflow/community/sofya/__init__.py - backend/packages/harness/deerflow/community/sofya/tools.py Implements web_search_tool and web_fetch_tool using httpx. API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY env var. Follows the same interface and output shape as the existing ddg_search and serper providers, including the max_results parameter with config override and the structured "No results found" error. - backend/tests/test_sofya_tools.py Unit tests covering API key resolution, config overrides, result mapping, time range, HTTP errors, empty results, and fetch failures. - config.example.yaml: add commented-out Sofya web_search and web_fetch examples alongside the other providers - .env.example: add SOFYA_API_KEY placeholder - backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and the environment variables * fix(sofya): honor caller max_results, validate search_depth, join time_range contract test - Caller-supplied max_results now wins; config is used only when the argument is omitted, matching GroundRoute. - search_depth is clamped to basic/snippets; an unsupported value logs a warning and falls back to basic. - Sofya added to the shared time_range schema contract test. * fix(sofya): cap per-result content so a search stays inline An unbounded search payload (up to 20 read pages) crossed the tool output budget middleware's externalize_min_chars threshold, which replaces the result list with a file reference. Cap each result's content at contents_max_characters (default 2000, 0 disables), matching Exa's config key. Five capped results stay under the 12000 char threshold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): list Sofya in the recency contract, coerce non-string content _clip subscripted its input, so a non-string content or description from the API raised TypeError instead of degrading. Coerce to text first, the way _sofya_post and _response_results guard the shapes around it. Also add Sofya to the Web Search Recency section in backend/AGENTS.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog web_fetch sliced its content the same way web_search did before the last push: a truthy non-string from the API passed the falsiness guard and then raised TypeError. Reuse _clip, keeping the `or ""` so empty content still reports "No content found". Also add sofya to the community provider inventory in packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs The ZH web_fetch tab list named five providers where EN names seven. Both tabs mirror their EN counterparts, so the two locales list the same web_fetch providers again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-08 10:50:03 +08:00
Wu Shuwen	364dad06aa	docs: update middleware contribution examples (#4945) ... * docs: update middleware contribution examples * docs: clarify middleware registration paths * docs: clarify middleware injection scope * docs: clarify middleware state updates * docs: clarify middleware state updates * docs: clarify middleware pipeline placement * docs: complete middleware order guidance * docs: align middleware guard conditions * docs(middleware): name runtime sanitization order * docs: pin middleware runtime order * docs: clarify middleware assembly paths * docs: clarify middleware anchor scope	2026-09-06 22:50:58 +08:00
hataa	aec7d73890	feat(knowledge): add read-only LightRAG retrieval, fixes #5208 (#5209)	2026-09-06 10:21:58 +08:00
Aari	0f7d8709d3	feat(sandbox): add controlled egress with approvals (#5152) ... * feat(sandbox): add controlled egress approvals * Apply batched suggestions from code review * fix(sandbox): harden restricted network policy * fix(sandbox): harden denied egress handling * fix(sandbox): isolate network proxy sidecar * chore: retry sandbox image smoke * fix(sandbox): close remaining network policy gaps * fix(sandbox): harden relay token rejection * fix(sandbox): fence incompatible policy replacement * fix(sandbox): replace containers across network modes * fix(sandbox): close remaining lifecycle gaps --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-04 23:46:57 +08:00
Michael	eebe909ebd	fix(agents): make the injected current-date timezone configurable (#5154) ... * fix(agents): make the injected current-date timezone configurable ## Why The date reminder injected into the lead and subagent prompts (DynamicContextMiddleware / SubagentDateContextMiddleware) was formatted with the server's local wall clock. DeerFlow containers default to UTC, so a user in Asia/Shanghai chatting in the 00:00-08:00 window was told that 'today' is the previous day - the model then reasons, plans, and date-stamps against the wrong day. ## What changed - _format_current_date() now reads the optional DEER_FLOW_DATE_TIMEZONE env var (IANA name, e.g. Asia/Shanghai) and renders the date in that zone. - Unset = unchanged server-local behavior; invalid names log a warning and fall back to server-local. - Documented the knob in config.example.yaml, the module docstring, and the DynamicContext entry in agents/middlewares/AGENTS.md. ## Surface area - [x] Agents / LangGraph - prompt-layer date context only; message shape and midnight-update behavior unchanged - [ ] Frontend UI / Backend API / Sandbox / Skills / Dependencies - [x] Default behavior change (opt-in via env var - no behavior change unless set) ## Bug fix verification - New tests: test_format_current_date_honors_configured_timezone (UTC 20:30 -> 2026-09-03 in Asia/Shanghai), test_format_current_date_defaults_to_server_local_without_env, test_format_current_date_invalid_timezone_falls_back. - Existing mocked-datetime tests pass unchanged (no env -> datetime.now() path). ## Validation - cd backend && python -m pytest tests/test_dynamic_context_middleware.py: 31 passed. - blocking_io/test_dynamic_context_middleware.py: 2 pre-existing abefore_agent failures reproduce identically on clean main (blockbuster os.listdir detection on this host); the other 2 pass. - ruff format + ruff check clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): avoid passing tz to datetime.now when no timezone is configured CI (backend-unit-tests shard 2) failed in test_tool_error_handling_middleware.py::test_subagent_chain_injects_date_without_memory_and_coalesces_for_strict_provider because its _FrozenDateTime.now() subclass override accepts no arguments, while _format_current_date() called datetime.now(None) even when DEER_FLOW_DATE_TIMEZONE was unset. - _format_current_date() now calls datetime.now() with no arguments unless a timezone is actually configured, preserving the exact legacy call shape for every datetime-subclass test fake. - The configured-zone path still calls datetime.now(tz) and converts via astimezone(tz). - Updated the no-env unit test to assert datetime.now() is called without arguments. Validation: python -m pytest tests/test_dynamic_context_middleware.py + the previously failing strict-provider test: 32 passed. ruff clean. * fix(agents): declare the effective current-date timezone in the assembly descriptor ## Why Maintainer review on the DEER_FLOW_DATE_TIMEZONE change (#5154): the knob is prompt-affecting, yet both DynamicContextMiddleware and SubagentDateContextMiddleware were invisible to the agent assembly descriptor - describe_middleware() fell back to {"probed": true} for unset, UTC, and Asia/Shanghai alike, so deployments that inject different dates shared one assembly fingerprint and release observers could not distinguish or audit the behavior change. ## What changed - Both middlewares now implement release_policy_parameters() -> dict[str, object], declaring {"current_date_timezone": <name>} as required by the module's middleware self-description contract. - The declared value is the normalized effective zone: a configured, valid DEER_FLOW_DATE_TIMEZONE is reported by its IANA key (ZoneInfo.key); otherwise the server-local zone is resolved to its IANA key when the platform exposes one and to its tzname label otherwise (fixed-offset hosts), with "UTC" as the final fallback. - Added both middlewares to _MIDDLEWARE_DECLARATIONS in backend/tests/test_middleware_release_policy.py so the existence check and the construct-and-canonical-hash check cover them. ## Verification - New tests: test_date_middlewares_declare_configured_timezone (Asia/Shanghai), test_date_middlewares_declare_utc_timezone, plus resolved-server-local assertions for the unset and invalid-env paths; both middlewares agree in every case. - cd backend && python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py: 70 passed. - Regression spot-check: tests/test_agent_assembly_descriptor.py, tests/test_tool_error_handling_middleware.py, tests/test_system_message_coalescing_middleware.py: 102 passed. - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): stabilize the declared date timezone and simplify the formatting path ## Why Follow-up review on #5154 (willem-bd). The release-policy declaration added in 884cec4b resolved the observability gap but pinned far less identity than its docstrings claimed, and the formatting path carried a production no-op. ## What changed - The declared label is now stable and unambiguous: a configured, valid DEER_FLOW_DATE_TIMEZONE is reported by its IANA key; without one, the server-local zone is resolved to a real IANA key from the TZ env var or the /etc/localtime symlink (Linux/macOS); when no key is recoverable (Windows, stripped containers) the declaration falls back to a stable `server-local(+-HH:MM)` sentinel carrying the current UTC offset. It never reports a bare abbreviation - datetime.now().astimezone() yields only a fixed-offset timezone whose tzname (e.g. CST, EST/EDT, CET/CEST) is ambiguous or DST-churns, which the assembly descriptor docstring says must not happen. - Dropped the redundant astimezone(tz) in _format_current_date(): datetime.now(tz) already returns the instant expressed in tz. The configured-zone test now fakes datetime.now(tz) semantics (the fixed instant converted into the requested zone) instead of relying on that conversion. - Documented why the knob is an env var, not a config-schema field: it is read at runtime by both date-context middlewares so an operator can point a container at another zone without mounting a config.yaml (module docstring + config.example.yaml note). - AGENTS.md: fixed the glued DynamicContext sentence (missing separator). - Added tzdata>=2025.1 to the harness runtime dependencies (with uv.lock) so ZoneInfo works on stripped containers / Windows without an OS zone database. ## Verification - New tests: test_server_local_timezone_name_reads_tz_env, test_effective_timezone_sentinel_uses_offset_when_local_zone_is_not_resolvable; reworked test_format_current_date_honors_configured_timezone to exercise the real datetime.now(tz) path. - cd backend && python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py tests/test_agent_assembly_descriptor.py tests/test_tool_error_handling_middleware.py: 140 passed. - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): offload subagent date injection off the event loop ## Why Follow-up review on #5154 (willem-bd, P2): SubagentDateContextMiddleware.abefore_agent() called _inject() directly, so enabling DEER_FLOW_DATE_TIMEZONE could synchronously read the OS timezone database (or the tzdata wheel) on a cold cache - filesystem work on the async subagent execution path whenever no assembly observer resolved the zone first. ## What changed - SubagentDateContextMiddleware.abefore_agent() now offloads the injection via asyncio.to_thread with the same bounded timeout DynamicContextMiddleware uses (issue #3402); on timeout it logs and skips the date update for that run instead of blocking the loop. - Narrowed the exception handling in _date_timezone() and the TZ-env branch of _server_local_timezone_name() to configuration-shaped failures (ZoneInfoNotFoundError / ValueError / OSError). Previously a blanket `except Exception` also swallowed BlockingError raised by the blocking-I/O regression gate, mislabeling a loop-blocking call as an invalid timezone and silently degrading to server-local - which made the new regression anchor useless. Other exceptions now propagate. ## Verification - New blocking-I/O regression anchor (backend/tests/blocking_io/test_subagent_date_context_middleware.py): drives a real create_agent graph under the strict Blockbuster gate with the knob enabled and asserts the date reminder is injected. Verified it fails (BlockingError) when the offload is reverted and passes with it in place. - python -m pytest tests/blocking_io/test_subagent_date_context_middleware.py: 1 passed. The two pre-existing os.listdir failures in tests/blocking_io/test_dynamic_context_middleware.py reproduce unchanged on this host (same as clean main). - python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py tests/test_tool_error_handling_middleware.py tests/test_agent_assembly_descriptor.py: 139 passed; the single ToolReceiptMiddleware-ordering failure reproduces with the change stashed (local extensions registry, unrelated to this PR). - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): read the direct /etc/localtime symlink target for the zone key ## Why Follow-up review on #5154 (willem-bd, P2): on macOS, /etc/localtime commonly points to /var/db/timezone/zoneinfo/<zone>, but Path.resolve() follows that directory's own symlink and yields a versioned path such as /private/var/db/timezone/tz/2026c.1.0/zoneinfo/Asia/Shanghai, which matched no configured prefix. The server-local resolution then returned None and the assembly descriptor fell back to a server-local(+HH:MM) sentinel even though the IANA key was available - conflating zones that share an offset and making DST-based fingerprints unstable. ## What changed - _server_local_timezone_name() now reads the direct symlink target via os.readlink("/etc/localtime") instead of Path.resolve(), so macOS' unversioned zoneinfo path is seen as-is and its IANA key is preserved. - The zone key is taken from whatever follows the last "/zoneinfo/" segment, which also handles Apple's canonical versioned path when a direct target already carries it, and relative targets are normalized against /etc. - Removed the now-unused Path import and the fixed zoneinfo prefix tuple. ## Verification - New tests: test_server_local_timezone_name_reads_direct_macos_symlink_target, test_server_local_timezone_name_reads_apple_versioned_symlink_target, and test_server_local_timezone_name_normalizes_relative_symlink_target. - python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py tests/test_agent_assembly_descriptor.py: 105 passed (75 after re-running the first two on the merged main). The blocking subagent anchor still passes; the two pre-existing os.listdir blocking failures on this host are unchanged. - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-04 23:39:31 +08:00
Tu Naichao	ae82f426bf	fix(summarization): stop fraction triggers from crashing the agent build (#4901) ... * fix(summarization): resolve fraction triggers from declared context_window, degrade instead of crashing the agent build A fraction trigger/keep clause requires profile["max_input_tokens"], which any third-party OpenAI-compatible model lacks, so SummarizationMiddleware construction raised ValueError out of create_summarization_middleware and failed the whole agent build (#3103). - factory: translate a declared model context_window into the langchain profile (metadata-only, never reaches the provider payload); explicit caller/override profiles win - summarization factory: drop unusable fraction trigger clauses (absolute clauses survive), fall a fraction keep back to the messages default, and disable compaction with an actionable warning only when no usable trigger clause remains — the agent build never dies from summarization config - docs: config.example.yaml, ModelConfig.context_window, summarization.md * refactor(summarization): share the default keep constant with the fraction fallback The fraction-keep degradation fallback hardcoded ("messages", 20), duplicating SummarizationConfig.keep's default_factory literal. Move the value to a shared DEFAULT_KEEP constant so the two cannot drift apart. * fix(summarization): keep trigger-null + fraction-keep constructing after degradation A trigger of None with a fraction keep hit the all-clauses-dropped branch (has_usable_trigger=False) and disabled compaction, and the accompanying warning claimed configured triggers were all fraction-based when none were configured. Only report nothing-usable when trigger clauses actually existed; trigger:null keeps constructing the never-firing middleware with the degraded keep, matching its behavior outside the degradation path. * fix(summarization): address review — keep manual compaction, validate ContextSize, pin wiring Review follow-ups on #4901: - When every configured trigger is a dropped fraction clause, keep constructing the never-firing middleware (trigger=None) instead of returning None: manual /compact runs with force=True and never consults trigger clauses, so it must keep working for a profile-less model rather than reporting 'compaction is disabled'. The warning now says auto-compaction will not fire while manual compaction remains. - ContextSize gains a config-load validator: fraction values must be in (0,1] (a percent-style 80 instead of 0.8 previously produced a threshold the context could never reach — a silently inert trigger), absolute values must be positive. - New un-monkeypatched integration test pins the shipped wiring (context_window declared -> real factory attaches profile -> fraction clause survives -> middleware constructs), which the stubbed middleware-side tests and kwarg-capturing factory-side tests each stopped short of. - Docs (summarization.md + config.example.yaml) clarify that the fraction resolves against the summary/anchor model's context_window (summarization.model_name when set, else the run model), including the mismatch caveat for a larger-window summary model. * fix(summarization): reject non-finite ContextSize values at config load YAML .nan / .inf pass pydantic's float parsing, and nan <= 0 is False, so the positivity check alone let them through as dead thresholds (count >= nan is always False) — the same silent-inert-trigger class the range validator was added to close. Guard with math.isfinite first, consistent with the existing non-finite guards on mem0 timeout_seconds and poll_after_seconds. * fix(summarization): merge context_window into inferred profile, require whole message counts - construct the model first, then merge max_input_tokens into the provider-inferred langchain profile: passing profile= to the constructor replaced the whole inferred metadata (tool_calling, structured_output, io capabilities, output limits) with the single key. An explicitly configured profile is still never clobbered. - reject non-integral ContextSize values for type=messages at config load: langchain slices the message list with them, so a float index raised TypeError mid-compaction. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-03 17:05:09 +08:00
hataa	08b27aef73	feat(auth): make login rate-limit parameters configurable, fixes #5108 (#5110) ... * feat(auth): make login rate-limit parameters configurable, fixes #5108 Add auth.local.max_login_attempts (default 5) and auth.local.lockout_seconds (default 300) so operators can tune the per-IP login throttle: raise the ceiling for shared-egress-IP offices behind proxies/NAT, or tighten it for stricter posture. Policy is live-read per call (matching the _local_registration_enabled precedent), so a config reload applies without a Gateway restart; raising the threshold mid-lockout immediately unblocks affected IPs. Review feedback addressed (willem-bd): - Only FileNotFoundError falls back to the hardcoded defaults; a malformed config propagates, mirroring _local_registration_enabled, so an operator who tightened the policy never silently gets the more permissive defaults. - _check_rate_limit looks up the record before resolving the policy, so a clean IP pays zero config reads (get_app_config re-hashes config.yaml per call and login_local is an unauthenticated async endpoint). Bumps config_version to 39 in config.example.yaml and the Helm chart (values.yaml + README example) so the chart drift check stays green. * fix(auth): reject max_login_attempts=1 and honor live lockout_seconds for active lockouts * fix(auth): close live-policy state gaps in login throttle (resurrection, count reset, broken-config verification) * fix(auth): commit evaluated lockout duration on decreases too, preventing raise-resurrection * test(auth): pin broken-config fail-closed sequence through the login route * fix(auth): sweep expired locks by stored sentence and keep policy reads off the event loop * fix(auth): re-read throttle record after the policy-resolution yield point --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-02 19:16:56 +08:00