mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 13:16:24 +00:00
131 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
IronMurphy
|
65ed502e54
|
fix(release): refresh backend/uv.lock when bumping the version (#5859)
* fix(release): refresh backend/uv.lock when bumping the version `scripts/bump_version.sh` rewrote backend/pyproject.toml, frontend/package.json and the Helm chart's `version`/`appVersion`, but `backend/uv.lock` also records the root package version (uv keeps its PEP 440 form, so `2.1.0-rc0` is stored as `2.1.0rc0`). The documented release step left the lock pinning the previous version, and that is not cosmetic drift: `uv lock --check` (lint CI) fails on a stale lock and `uv sync --locked` (e2e / plugin workflows) refuses the tree, so the release commit could not pass CI. RELEASING.md only mentioned the manual `cd backend && uv lock` step for release candidates. The script now refreshes the lock via `uv lock` and refuses to start without `uv` — it exits before editing anything rather than leaving a half-bumped tree whose lock CI rejects. `uv lock` stays the authority for the normalized prerelease form; the lock diff for a bump is a single line. RELEASING.md and AGENTS.md now describe the lock as part of the bump. Tests: backend/tests/test_bump_version_lockfile_sync.py (red on main, green here) pins both halves — the lock moves with the sources, and a missing uv is an error instead of a silent no-op. * docs(changelog): record the release bump lockfile fix (#5859) --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
xiaodu55
|
64e499be98
|
feat(memory): redact queued memory payloads (pii_redaction follow-up) (#5577)
* feat(memory): redact queued memory payloads (pii_redaction follow-up) Closes the remaining vector from #3190 (memory storage): request-scoped redaction leaves originals in thread state, and MemoryMiddleware queued that raw conversation into the durable extraction buffer — so the extraction model's input and the persisted facts carried raw PII even with pii_redaction.enabled. - MemoryMiddleware gains pii_redaction_config (same pattern as summarization / durable-context / title); the payload returned by _resolve_add_args() is redacted at the enqueue boundary before add()/aadd(), covering both after_agent and aafter_agent - Reuses the shared redact_text() seam from #5527 via active_pii_detectors/_Redactor/_redact_content; the buffered payload itself becomes clean, not just the extraction model's view - Wired at the lead-agent assembly site - Tests: queue-payload redaction enabled/disabled, detector toggles, message immutability, token stability across turns * fix(memory): reserve existing placeholder indices across the queued batch Review round 1 on #5577: thread state mixes raw user turns with already-redacted tool results and summaries, so a token in a later message must not collide with a new value in an earlier one. Pre-scan the whole queued batch with redactor.reserve() before allocating. Also extend the two MemoryMiddleware stubs in test_lead_agent_model_resolution.py to accept the new pii_redaction_config kwarg (CI backend-unit-tests red). * fix(memory): redact the compaction-triggered memory_flush_hook queue too Review round 2 on #5577: with memory + summarization enabled, memory_flush_hook() queues list(event.messages_to_summarize) directly into MemoryManager.add_nowait(), bypassing MemoryMiddleware entirely — and compaction removes those messages from state right after, so the after-agent redaction can never repair a raw batch queued there. Extract the batch redaction (reserve pre-scan + rebuild) into a shared redact_queued_messages() used by both MemoryMiddleware's enqueue boundary and the hook; the hook gains an optional pii_redaction_config wired via functools.partial at the summarization factory (subagents use skip_memory_flush=True and are unaffected). Regression covers the hook path enabled/disabled/absent. * test(summarization): unwrap the partial in the factory hook identity check edd916e wrapped memory_flush_hook in functools.partial to carry the pii_redaction config, but missed committing the matching update to test_factory_attaches_memory_flush_hook_by_default — a functools.partial is not equal to its wrapped function, so the identity assertion failed deterministically (caught by static review on #5577). * fix(memory): cover tool-call arguments and lenient text blocks in queued payloads Review round 4 on #5577 — two reproduced gaps still allowed raw PII into memory payloads with pii_redaction.enabled=true: - OpenViking retains the full message object, so parsed tool_calls args and provider-format arguments in additional_kwargs bypassed the content-only redaction; queued payloads now redact tool-call args recursively (and reserve them in the pre-scan). - DeerMem's format_conversation_for_update reads p.get("text") of any dict block, not only type=="text" — _redact_content now redacts every string-valued text field so a lenient consumer cannot bypass it. Both repro shapes are pinned as regressions. * fix(pii): value-derived placeholders for stable capture identity Review round 5 on #5577: sequential allocation is order-dependent — the same unchanged message re-redacted in a later batch can get a different token, which flips OpenViking's content-signature and replays already recorded turns into persistent memory sessions. Placeholders are now a pure function of the matched value (sha256-6 per category): identities stay stable across batches, enqueues, seams, and backends with no shared state and no stored mapping. - _Redactor becomes stateless; the reserve machinery (index pre-scans, _PLACEHOLDER_PATTERN) is removed along with its call sites - redact_texts keeps its signature, now a simple per-text redaction - reserve-semantics tests are superseded by cross-enqueue identity tests (two successive captures where the second batch carries higher existing placeholders); the compiled-graph echo regex matches the new format * fix(pii): 128-bit placeholder digests; redact tool-argument keys Review round 6 on #5577: - [P2] The 24-bit truncation collided distinct identities (reproduced: contact3513@ / contact3727@example.com both rendered [EMAIL_575dbb]). Retain 128 bits of the digest so separate people stay separate at any realistic volume; the compiled-graph echo regex and the AGENTS.md entry follow the new format. - [P2] _redact_strings redacted values but preserved mapping keys, so {"contacts": {"alice@example.com": "manager"}} kept the raw email as a key. Redact string keys along with values; deterministic 128-bit tokens make distinct-key collisions a hash-collision non-event, and a redacted key equal to another key merges the entries. Regression pins the email-keyed arguments shape. * docs(agents): trim PiiRedactionMiddleware entry; drop stale reserve wording The chain grew past the 98304-byte AG002 hard limit again (main growth plus the round-6 format example). Trim the entry to fit, and replace the stale reserve-indices wording with the value-derived semantics that replaced it. * docs(agents): shrink PiiRedactionMiddleware entry below the chain budget main moved (#5609) and the base comparison flipped the nearly-over chain back to an AG002 error (98441 > 98304). Trim the entry to the facts that are not documented elsewhere: seams, allowlist, order, enqueue boundary. * fix(pii): letters-only value-derived tokens; wire the factory SDK path Review round 7 on #5577: - [P2] Hex digests carry digit runs that later digit-anchored detectors re-scan, corrupting minted tokens into nested placeholders (reproduced: user280@example.com -> [EMAIL_…[PHONE_…]…], ~1/140 emails). The 128-bit digest is now encoded over a-z, so a minted token survives the full pinned detector order. - [P2] create_deerflow_agent (the SDK path) constructed MemoryMiddleware and DurableContextMiddleware without pii_redaction_config; the knob is now an explicit optional parameter threaded through _assemble_from_features (the factory reads no config files by design, so SDK callers pass it explicitly). - [Nit] hoist the Command rebuild comprehension out of the per-message loop in _redact_result. - [Nit] drop the dead _iter_tool_calls helper and Iterator import. * test(factory): pin pii_redaction_config wiring; document the parameter Review round 8 on #5577: - [P3] test_create_deerflow_agent.py had zero coverage of the pii_redaction_config parameter; two regressions now assert that create_deerflow_agent(pii_redaction_config=cfg) builds a MemoryMiddleware and a DurableContextMiddleware carrying cfg, and that the default is None (redaction off). - [Nit] document the parameter in the create_deerflow_agent docstring: None leaves redaction off; a middleware= takeover bypasses the assembly that would apply it. * fix(pii): HMAC-scoped placeholder tokens; install model-bound redaction in the SDK chain Review round 10 on #5577: - [P2] sha256(category||value) is an unkeyed, globally deterministic fingerprint: a provider or memory backend can confirm candidate values offline and link identical values across users and threads, contradicting the documented no-re-identification guarantee. Placeholders are now HMAC-derived over a deployment-scoped token_secret (new config field); the docstring documents the linkability scope for both modes. - [P2] _assemble_from_features() never appended PiiRedactionMiddleware, so the SDK knob enabled only the downstream seams while the primary model still received raw content. The middleware is now appended in the same relative position as the lead-agent base chain when the gate is on, with a model-call regression through the assembled SDK chain. * docs(pii): document token_secret; update the placeholder format examples Follow-up to the round-10 review on #5577: - config.example.yaml documents the new token_secret knob: a deployment-scoped secret makes placeholders HMAC-derived and only linkable within one deployment; leaving it unset keeps publicly computable fingerprints (documented trade). - the PiiRedactionConfig class docstring's placeholder example now matches the shipped format (128-bit HMAC digests over base-26 letters). * docs(pii): update the module docstring's placeholder format example The module docstring still showed the sequential [EMAIL_1] example; align it with the shipped value-derived format (128-bit HMAC digests over base-26 letters). * fix(pii): require a non-empty token_secret when redaction is enabled Review round 11 on #5577: with the secret optional, the default enabled configuration minted empty-key HMAC digests — publicly computable, globally linkable fingerprints of the raw values, confirmable by offline guessing for low-entropy identifiers such as phone numbers. - PiiRedactionConfig now rejects enabled=true without a non-empty token_secret (minimum 16 characters); the unkeyed mode is gone entirely. - The request-scoped redactor in _process_request now carries the configured key: it built its redactor without it, so user-message placeholders silently diverged from every other seam for the same value. The empty-key guard in _placeholder_token is the backstop that keeps every path keyed. - Module/config docstrings and config.example.yaml no longer describe the placeholders as irreversible or the secret as optional; the stale reserve-semantics wording and the "memory path not covered" note are corrected. - Regressions: validation errors for missing/blank/short secrets, and the user-message boundary minting the same keyed token as redact_text. * fix(memory): redact original_user_content on queued memory payloads Review round 12 on #5577: UploadsMiddleware preserves the raw user turn in additional_kwargs.original_user_content, but redact_queued_messages rewrote only content, tool_calls, and provider-format tool_calls — the queued copy still carried the raw value, so the memory queue and backend received raw PII with redaction enabled. The shared helper now redacts the provenance string field on the queued copy (same batch redactor, merged with the existing additional_kwargs rewrite, rebuilt via model_copy — thread-state messages stay untouched), covering both the enqueue boundary and the compaction flush hook. Regressions pin the redacted provenance field through both paths and the untouched original. * fix(memory): redact human_input_response on queued memory payloads Review round 13 on #5577: clarification replies keep the raw user answer in additional_kwargs.human_input_response.value (DeerMem reads that mapping via its read_human_input_response mirror), but redact_queued_messages rewrote only content, tool_calls, and original_user_content — the queued copy still carried the raw value into the memory queue and backend. The shared helper now rebuilds that mapping with value redacted (same batch redactor) while preserving the protocol fields (version/kind/source/ request_id/option_id/response_kind), merged with the existing additional_kwargs rewrite and rebuilt via model_copy so thread-state messages stay untouched. Regressions pin the redacted value, preserved protocol fields, the untouched original, and the flush-hook path. * chore(config): bump config_version to 47; generate token_secret on upgrade Review round 13 on #5577: existing deployments can legitimately sit at config_version 46 with pii_redaction.enabled: true and no token_secret; the mandatory-secret validator would fail their startup while config.example.yaml still said 46, so the upgrade checker considered them current and emitted no migration warning. - Bump config_version to 47 so the outdated-config warning fires for v46. - Add a v47 migration to scripts/config-upgrade.sh: when pii_redaction is enabled and token_secret is missing/blank, generate a random deployment- scoped secret (secrets.token_urlsafe) and persist it, so the upgraded deployment starts under the mandatory validation. - Regressions: a v46 enabled config is flagged outdated against the real example; the upgrade run generates a >= 16-char secret and the result passes AppConfig validation; a disabled config gets no secret. - Also switch the two v41 script tests from a hardcoded "bash" to SCRIPT_BASH/skipif, matching the documented Windows constraint and the neighboring tests (hardcoded bash resolves to the WSL stub there). * chore(helm): sync chart config_version to 47 Review round 14 on #5577: check_config_version.sh fails whenever the chart's embedded config_version lags config.example.yaml, and the chart workflow is triggered by config.example.yaml — so the v47 bump left this head failing that check. Update the values.yaml config block and the README example to 47; scripts/check_config_version.sh now passes. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
bcaf6db36a
|
feat(models): add declarative reasoning capability contract (#5780)
* feat(models): add declarative reasoning capability contract (#5073) Introduce an optional per-model `reasoning:` block beside the legacy `supports_thinking` / `supports_reasoning_effort` booleans. It declares thinking availability (unsupported | optional | required), the accepted effort values with aliases, default, and serialization path, the payload dialect, and the reasoning-history requirement. When present the booleans are derived from it and contradictory profiles fail at config load. `deerflow.models.reasoning` normalizes legacy and declared profiles into one immutable contract and resolves every caller's generic request against it. `create_chat_model` is the single enforcement point: required-thinking models never receive a synthesized disable payload, unsupported effort values never reach the provider, and the dialect synthesizes the on/off payload when no template exists. Legacy profiles keep the historical path byte-for-byte. The lead agent and subagent descriptor resolve the same policy first so run metadata reports effective values. `/api/models` and `DeerFlowClient` project the contract as `reasoning`; the frontend derives mode clamping and effort choices from it through `core/models/reasoning.ts`, and user preferences accept provider-specific effort tokens. The setup wizard's Z.AI GLM-5.3-Flash profile migrates to the contract, restoring its low/high/max effort control. * docs(models): move the reasoning contract guidance into models/AGENTS.md backend/AGENTS.md sits within 46 bytes of the 96 KiB effective-chain hard limit for the middlewares, runtime, sandbox, and subagents guides, so the new section pushed scripts/check_agent_guidance.py over it. The models guide is the nearest file for the factory and the contract module, and its chain has ample room. * fix(models): address review on the reasoning contract - vLLM dialect: the enable path now mirrors whichever chat_template_kwargs switch the template declares, like the disable path already did, so a template using the older `thinking` key no longer also grows `enable_thinking`. - Validate operator-supplied effort values at `effort.path` inside `when_thinking_enabled`, `when_thinking_disabled`, and the `thinking` shortcut against the contract, not only the profile-level value; they are forwarded when the caller chooses nothing, so the "unknown values never reach the provider" guarantee now holds for them too. - Constrain `effort.path` to dotted identifiers and reject single-segment paths that would replace a whole mapping (`extra_body`, `thinking`, `model_kwargs`, `default_headers`, `default_query`). - Z.AI GLM-5.3-Flash profile: `default: high` instead of `max`, because the default also governs summarization, title generation, and subagent calls that never choose an effort; `max` stays selectable. Documented in the config example, the configuration manual, and the design note. * fix(models): close reasoning contract review gaps * fix(models): preserve native Ollama reasoning config * fix(models): keep native reasoning strings and project reasoning into the descriptor - `ModelConfig.reasoning` also accepts the level string ChatOllama takes for gpt-oss style models (`low` / `medium` / `high`); the factory forwards both the boolean and the string on the legacy path, so profiles that passed `reasoning: high` through before the contract field existed keep loading and keep sending it. - The assembly descriptor no longer excludes `reasoning` from `model_parameters`: the native ChatOllama value and a declared contract's dialect/history/effort change the request payload, so they now move the assembly fingerprint. - e2e: the sidecar spec pinned the pre-contract body where a model without effort control still received `reasoning_effort: minimal`; the preset is now dropped for such models (the Gateway discarded it anyway), so the assertion expects the key to be absent. |
||
|
liunianxuxie
|
4ccf4e1729
|
fix(setup): accept successful Apple Container image pulls (#5804) | ||
|
Keswyk Cheung
|
731e8c6eee
|
feat(memory): bulk-load sample for registered users (#4066)
* feat(memory): bulk-load sample for registered users * fix(memory): harden all-users sample loader --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
laundry
|
1ffe82eb65
|
fix(setup): run pre-commit through uv and clarify Node advice (#5767) | ||
|
spud
|
73e2be20e1
|
fix(helm): align provisioner state root for PVC mounts (#5625)
* fix(helm): align provisioner state root for PVC mounts * test(helm): pin sandbox PVC subPath contract |
||
|
Zheng Haoran
|
2529260c19
|
fix(doctor): validate CLI credential contents (#5567)
* fix(doctor): validate CLI credential contents * fix(doctor): keep provider results when CLI auth state is undecodable A credential file holding non-UTF-8 bytes made read_text raise UnicodeDecodeError, which _load_json_object did not catch. check_llm_auth wraps the whole model loop in one handler, so a single bad file replaced every provider result with one generic failure and dropped the provider-specific fix hints. Catch it alongside the malformed-state cases and pin the behaviour with a two-model regression test. Also cover the acceptance branches the change claims but did not test (malformed JSON, non-object JSON, a directory as the auth path, blank tokens, non-numeric expiresAt) and record credential_loader.py as the source of truth for the mirrored rules, including the one place doctor is deliberately stricter. |
||
|
xiaodu55
|
859b105b40
|
fix(docker): add 'make prod-logs' entry point and hint when dev logs are empty (#5538)
* fix(docker): add 'make prod-logs' entry point and hint when dev logs are empty `make up` starts the production stack (deploy.sh: project `deer-flow`, docker/docker-compose.yaml) while `make docker-logs` tails the dev stack (project `deer-flow-dev`, docker-compose-dev.yaml), so after `make up` it printed nothing (#5529), and no make entry point showed production logs at all. - scripts/docker.sh logs gains `--prod`: targets the stack deploy.sh started, passes --env-file ../.env when present, and exports the same interpolation defaults deploy.sh exports before every compose call — without them the production volume specs fail to parse on checkouts without .env. - dev-only `logs` with no running containers now prints a hint pointing at `make prod-logs` instead of staying silent. - Makefile gains `prod-logs`, listed under Docker Production Commands. New tests cover production targeting and the empty-state hint; they are red on unfixed main and green here. Verified live: with a deer-flow redis running, `logs --prod --redis` streams its logs. Fixes #5529 * fix(docker): append --env-file after compose detection rebuilds COMPOSE_CMD compose_preflight() probes the Compose binary and rebuilds COMPOSE_CMD, so an --env-file appended before it was silently dropped. Append it after preflight instead, and drive the regression test through the real detection path (stub Docker Compose version v5.3.1, not require_compose_version) so the append cannot regress silently. Reviewed-in: #5538 |
||
|
zhangwei-way
|
b6503e9a35
|
feat(knowledge): add per-message RAGFlow retrieval scope (#5238)
* feat(knowledge): integrate RAGFlow retrieval and management * test(knowledge): cover merged listing tool * feat(knowledge): add per-message retrieval scope * chore(docs): remove unrelated document * docs(knowledge): add interaction screenshots * feat(knowledge): simplify scope selector trigger * docs(knowledge): refresh selector screenshot * feat(knowledge): defer standalone management * docs(knowledge): show chat-only scope UI * fix(knowledge): honor scope on clarification replies * fix(knowledge): harden scoped replay validation * docs(knowledge): clarify replay scope precedence * fix(knowledge): keep provider settings on tools * fix(config): preserve tools-only knowledge settings * fix(knowledge): submit custom assistant identity * refactor(knowledge): trim PR scope changes * fix(knowledge): sanitize document scope display * feat(knowledge): enable scope selection in main chat * fix(knowledge): emphasize active scope icon without button frame * fix(knowledge): close context scrubbing and refresh e2e checks * fix(knowledge): preserve idempotent canonical retries * fix(knowledge): accept promptless conversation runs * style(knowledge): format backend regression tests * chore(knowledge): trim PR scope and fix frontend format * fix(knowledge): remove shared-scope notice * fix(knowledge): remove scope persistence notice * docs(knowledge): include main chat in catalog scope * fix(knowledge): preserve scope recovery and upgrades * fix(config): preserve LightRAG knowledge upgrades --------- Co-authored-by: foreleven <for-eleven@hotmail.com> |
||
|
liunianxuxie
|
53f2a73d23
|
fix(setup): honor sandbox image in BOM-prefixed configs (#5515)
* fix(setup): honor sandbox image in BOM-prefixed configs * fix(setup): normalize CRLF and preserve captured pull arguments |
||
|
liunianxuxie
|
e19c37d813
|
fix(setup): detect optional extras in BOM-prefixed configs (#5504) | ||
|
liunianxuxie
|
42629c8ac6
|
fix: detect browser extra regardless of tool field order (#5456)
* fix: detect browser extra regardless of tool field order * test: address browser extra detection review feedback |
||
|
Dan Caldr
|
f66cb8e376
|
fix(deploy): prevent Git Bash path conversion of default docker socket on Windows (#5400) (#5402)
* fix(deploy): prevent Git Bash path conversion of default docker socket on Windows (#5400) * fix(deploy): resolve docker socket from dotenv and assert custom socket preservation * docs: trim backend guidance to satisfy inherited size budget --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Dan Caldr
|
ce635b7de6
|
fix(docker): allow aio DooD socket preflight on Windows Git Bash (#5370) (#5371)
* fix(docker): allow aio DooD socket preflight on Windows Git Bash
* test(deploy): add regression tests for deploy.sh DooD socket preflight
* fix(docker): restrict Windows DooD socket bypass to default path
Limit the Windows socket bypass in docker.sh and deploy.sh to the default /var/run/docker.sock path, and add regression tests ensuring custom missing socket paths are rejected.
* test(docker): skip unreachable socket controls on hosts with live docker socket
Add @pytest.mark.skipif on Path('/var/run/docker.sock').is_socket() to prevent false test failures on daemon-live hosts.
|
||
|
Parthiban Sivakumar
|
a2011996d8
|
fix(scripts): detect the ollama extra from configured models (#5318)
* fix(scripts): detect the ollama extra from configured models make dev synced without --extra ollama, uninstalling langchain-ollama from a working setup. Same failure #2754 hit with postgres; ollama predates the detector added in #2767 and never got a rule. * fix(deps): declare the ollama extra on backend, pin model use: matching Review follow-ups. `ollama` was the only extra in the detector's map that the root `backend` project did not declare, so any consumer syncing without `--all-packages` failed outright: $ cd backend && uv sync --locked --extra ollama error: Extra `ollama` is not defined in the project's `optional-dependencies` table `serve.sh` and `docker/dev-entrypoint.sh` both pass `--all-packages` and were unaffected, but `backend/Dockerfile` does not, and `config.example.yaml` documents `UV_EXTRAS` as an image build-arg — so `UV_EXTRAS=ollama docker compose build` would have hard-failed on a value this branch makes first-class. Declaring `ollama = ["deerflow-harness[ollama]"]` alongside the other delegating extras closes that, and `uv.lock` is regenerated to match. The `use:` match also accepted any nesting depth inside `models:`, so a `use` in a sub-mapping was read as the model's provider: models: - name: doubao use: deerflow.models.patched_deepseek:PatchedChatDeepSeek when_thinking_enabled: use: langchain_ollama:ChatOllama That yielded `--extra ollama` despite the model's own provider pointing elsewhere, and `when_thinking_enabled` appears fifteen times in config.example.yaml, so the shape is common rather than contrived. Pin matching to the list item's own key indent, mirroring how `section_value()` pins `child_indent` and documents deeper nesting as ignored on purpose. Covered by a regression test that fails on the looser parser. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(deps): drop unrelated standard-aifc marker churn from uv.lock Review follow-up. Regenerating the lock with a newer local uv (0.11.19) also rewrote the `standard-aifc` entry, adding `python_full_version >= '3.13'` markers to its `audioop-lts` and `standard-chunk` dependencies. Unrelated to this change, so restore upstream's lines and keep the lock diff to the `ollama` extra. `uv lock --check` passes with these lines under both the CI- and Dockerfile-pinned uv 0.11.1 and uv 0.11.19. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scripts): detect ollama in unindented model lists and after nested sequences Review follow-ups on the models parser. `yaml.safe_dump` — used by the setup wizard (`make setup`) and scripts/config-upgrade.sh — writes list items unindented: models: - name: qwen3-local use: langchain_ollama:ChatOllama The parser treated any column-0 line as the end of the `models:` section, so the first model ended it and nothing was detected. That is the layout new users get from the recommended setup path, so `make dev` still synced without `--extra ollama`. Model entries are now recognised before the section-end test, the same ordering `tools_include_name()` already uses for the unindented tools list (#4367). Separately, every sequence item reset the key indent, including items inside a model option. With `stop:` / `- END` before `use:`, the key indent jumped to the nested item's and the model's own `use` was skipped, so detection depended on key order within the model. The first sequence item under `models:` now fixes the model-list indent; only items at that indent start a model and set where its keys sit. Nested sequence items and deeper mappings are skipped without moving it. Regression tests cover the real setup-wizard output via `build_minimal_config()`, a hand-written unindented list, and `use:` after a nested `stop:` list — all three fail on the previous parser — plus a guard that a `- use:` nested inside a model option is still ignored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
dong
|
f09e824460
|
fix(scripts): prefer Windows .cmd shims for pnpm (#5305)
* fix(scripts): prefer Windows cmd shims for pnpm * fix(scripts): address pnpm Windows review feedback * docs: reduce inherited agent guidance size |
||
|
Parthiban Sivakumar
|
065f84f711
|
fix(doctor): skip tool checks when tools block is empty (#5301)
* fix(doctor): skip tool checks when tools block is empty Follow-up to #5296, which fixed this for `models:`. The same defect remains for `tools:`: `.get("tools", [])` returns None when the key is present but empty, because the default only applies when the key is absent. Iterating that None raises TypeError, which the surrounding broad handler renders as a check result: ! web search configured ('NoneType' object is not iterable) ! web fetch configured ('NoneType' object is not iterable) ! web capture configured ('NoneType' object is not iterable) ! image search configured ('NoneType' object is not iterable) ✗ sandbox configured ('NoneType' object is not iterable) Line 476 is reached by all four web/image checks through the shared check_web_tool helper, and line 645 by check_sandbox. Unlike the models case, a default install does not hit this: `make config` ships ten real tool entries, so a user has to empty or comment out that block first. The web checks now fall through to their normal "no tool in config" warning and the sandbox check evaluates normally. Parentheses on the comprehension are for readability; `or` already binds correctly there. Regression tests use the commented-out `tools:` shape that reproduces the failure, matching the tests added in #5296. Fixes #5300 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(doctor): skip non-mapping tool entries, tighten regression tests Review follow-ups on the line this PR already changes. A `tools:` list holding a scalar (`tools:\n - web_search`) reached `t.get("name")` and raised AttributeError, which the broad handler rendered as the check result: ! web search configured ('str' object has no attribute 'get') That is the same leakage this PR removes for the null case, so it is fixed here rather than deferred. `check_sandbox` already guards the same way via `isinstance(tool, dict)`. The empty-tools test asserted that "NoneType" was absent from the detail, which pins the failure mode rather than the behaviour — it would still pass if the detail became some other internal error text. Both tests now assert the expected message directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(doctor): assert sandbox outcome instead of the failure mode Review follow-up, same class as the web-tool assertion fixed earlier in this PR. The sandbox regression test still asserted that "NoneType" was absent from the detail, which pins the failure mode rather than the outcome — it would keep passing if some other internal error text leaked out of the broad handler. On this config the path is deterministic: an empty `tools:` means no bash tool, so exactly one result. Assert the fields directly (`CheckResult` has no `__eq__`, so whole instances cannot be compared by value). Verified against `main`'s scripts/doctor.py, where the same config yields status=fail and detail="'NoneType' object is not iterable", so the new assertions are red there and green here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Parthiban Sivakumar
|
611801d5c0
|
fix(doctor): skip LLM checks when models block is empty (#5296)
`config.example.yaml` ships a `models:` key with every entry commented
out, so it parses as None rather than an empty list and the `[]` default
in `.get("models", [])` never applies. Iterating that None raised
TypeError, which the surrounding broad handler rendered as a check
result:
✗ LLM API key check ('NoneType' object is not iterable)
✗ LLM auth check ('NoneType' object is not iterable)
✗ LLM package check ('NoneType' object is not iterable)
Every fresh install hit this before configuring a model, turning one
actionable error into four and hiding the real "models configured" hint
behind internal exception text.
Fall back on a falsy value at the three iteration sites so the checks
return no results when nothing is configured. `check_models_configured`
gets the same treatment for consistency; it was already correct because
it tests truthiness rather than iterating.
The existing tests missed this because they use `models: []`, an
explicit empty list, which iterates fine. The added regression tests use
the commented-out shape that `make config` actually produces.
`make doctor` now reports 1 error instead of 4 on a fresh clone.
Fixes #5295
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Yusuf Gürdoğan
|
23bd76046a
|
feat(community): add Sofya web search provider (#5239)
* feat(community): add Sofya web search provider Add a community provider backed by Sofya (https://sofya.co). Its search endpoint returns the content of the result pages, not only their snippets, and its fetch endpoint returns a page as markdown. Both are plain JSON over HTTP, so this needs no extra Python package (uses httpx, already a dependency). Changes: - backend/packages/harness/deerflow/community/sofya/__init__.py - backend/packages/harness/deerflow/community/sofya/tools.py Implements web_search_tool and web_fetch_tool using httpx. API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY env var. Follows the same interface and output shape as the existing ddg_search and serper providers, including the max_results parameter with config override and the structured "No results found" error. - backend/tests/test_sofya_tools.py Unit tests covering API key resolution, config overrides, result mapping, time range, HTTP errors, empty results, and fetch failures. - config.example.yaml: add commented-out Sofya web_search and web_fetch examples alongside the other providers - .env.example: add SOFYA_API_KEY placeholder - backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and the environment variables * fix(sofya): honor caller max_results, validate search_depth, join time_range contract test - Caller-supplied max_results now wins; config is used only when the argument is omitted, matching GroundRoute. - search_depth is clamped to basic/snippets; an unsupported value logs a warning and falls back to basic. - Sofya added to the shared time_range schema contract test. * fix(sofya): cap per-result content so a search stays inline An unbounded search payload (up to 20 read pages) crossed the tool output budget middleware's externalize_min_chars threshold, which replaces the result list with a file reference. Cap each result's content at contents_max_characters (default 2000, 0 disables), matching Exa's config key. Five capped results stay under the 12000 char threshold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): list Sofya in the recency contract, coerce non-string content _clip subscripted its input, so a non-string content or description from the API raised TypeError instead of degrading. Coerce to text first, the way _sofya_post and _response_results guard the shapes around it. Also add Sofya to the Web Search Recency section in backend/AGENTS.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog web_fetch sliced its content the same way web_search did before the last push: a truthy non-string from the API passed the falsiness guard and then raised TypeError. Reuse _clip, keeping the `or ""` so empty content still reports "No content found". Also add sofya to the community provider inventory in packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs The ZH web_fetch tab list named five providers where EN names seven. Both tabs mirror their EN counterparts, so the two locales list the same web_fetch providers again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
pclin
|
fcb1c88e5e
|
fix(scripts): probe _pick_python candidates through env so make dev starts the frontend on Windows (#5181)
* bugfix #5179 * test: cover the env-aware _pick_python fallback from #5179 Follow the test_serve_nginx_stop.py extraction pattern: drive the real _pick_python from serve.sh against a stub-only PATH plus a mocked env. - python3 succeeds directly but fails through env -> python selected (red on main, green on this branch) - env rejects every candidate -> nonzero exit (also red on main) - healthy PATH with the real env -> python3 preferred, guarding against over-rejection MSYS/Git Bash hosts need the stub dir as an MSYS-style (/c/...) PATH entry, and bash diagnostics may arrive in the console code page, so the runner decodes output with errors="replace". |
||
|
Willem Jiang
|
eac028cca6
|
ci: preauthorize skill review waiver hashes (#5143) | ||
|
Willem Jiang
|
1af79c7bcf
|
fix(ci):resolve the skill_review errors (#5121)
* fix(ci):resolve the skill_review errors * fix(ci): split skill creator fixes from waiver rollout |
||
|
PeaceMaker-best
|
72ba661b84
|
feat(skills): install local skill archives (#5039)
* feat(skills): install local skill archives Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> * fix(skills): enforce upload limits before parsing * fix(nginx): scope skill upload limit to upload route * fix(nginx): harden skill upload proxy handling * fix(skills): improve archive upload feedback * fix(skills): address upload review polish --------- Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> |
||
|
早上肚子疼
|
b41354d75f
|
fix(scripts): invoke repo shell scripts through an explicit interpreter (#5031)
Recipes and scripts ran sibling shell scripts bare (./scripts/x.sh), so
any checkout that lost the executable bit -- zip/tarball download,
core.fileMode=false, non-POSIX filesystem -- failed with:
make: ./scripts/docker.sh: Permission denied
make: *** [Makefile:181: docker-start] Error 127
The tracked modes are already 100755, so chmod cannot fix it. Name the
interpreter instead: the POSIX branch of RUN_SHELL_SCRIPT (renamed from
RUN_WITH_GIT_BASH) now expands to $(BASH) rather than nothing, and the
five script-to-script call sites are prefixed with bash.
Fixes #2903
Co-authored-by: zaoshangduziteng <309590849+zaoshangduziteng@users.noreply.github.com>
|
||
|
Nan Gao
|
bf3e792a6a
|
feat(models): add GLM-5.3-Flash thinking workaround (#5074) | ||
|
zhang
|
23d8e4b3a3
|
feat(scripts): support skipping frontend build on make start (#5053)
* feat(scripts): support skipping frontend build on make start * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(scripts): validate skip-frontend-build before stop_all and format test --------- Co-authored-by: PoetryLin <PoetryLin@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Serply
|
4dbfe37ff3
|
feat(community): add Serply web search tool (#5023)
Add deerflow.community.serply.tools:web_search_tool, a Google SERP provider for the web_search slot that also covers Google News and Google Scholar through an optional `vertical` config option. Reads the key from api_key in config.yaml or SERPLY_API_KEY, clamps max_results to Serply's 1-100 range, and returns the same structured JSON errors as the Serper and Brave tools. Register the provider in config.example.yaml, scripts/doctor.py, scripts/wizard/providers.py, .env.example, backend/docs/CONFIGURATION.md, the en/zh tools.mdx provider tabs, and tools/AGENTS.md. Tests mock httpx. |
||
|
yong
|
846c716523
|
feat(search): add Tencent Cloud WSA provider (#5057)
* feat(search): add Tencent Cloud WSA provider * docs: restore README to upstream * docs: remove README changes from WSA provider PR * fix(doctor): validate Tencent WSA API key |
||
|
YZJF,YCDG,DJLY,ZZZB
|
851e76661b
|
fix(docker): don't abort Docker startup when .env is missing (#4956)
* fix(docker): create compose env files and keep Windows compose paths relative Windows Docker reports a generic file-not-found when env_file targets are missing, or when compose paths are doubled. Make docker-start copy .env examples and invoke compose with filenames relative to docker/. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(docker): make dev compose env files optional and repair test fixture Address review feedback on #4956. [P1] prepare_compose_env aborted before the mocked COMPOSE_CMD in test_compose_commands_set_deer_flow_root_before_compose, because the temp root had no compose file or .env examples. Seed them in the fixture so the preflight reaches the mock. [P2] .env is gitignored, so a fresh clone has none and a direct `docker compose -f docker/docker-compose-dev.yaml up --build` aborts on Windows before scripts/docker.sh can help. Mark the dev env_file entries `required: false` so a missing .env is not fatal, and document that direct Compose must be run from the repository root. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(docker): declare Compose 2.24 floor and keep non-start commands read-only Address the second review round on #4956. - Document Compose >= 2.24 (CONTRIBUTING, README, compose header) and fail early from make docker-start with an actionable message; probe both `docker compose` and the hyphenated `docker-compose` binary. - Document DEER_FLOW_ROOT for direct Compose callers (bash + PowerShell); leave the variable without a $PWD fallback because PowerShell/cmd do not export it. - Split prepare_compose_env: compose_preflight is shared and read-only; ensure_env_files runs only from start. - Expand tests for version boundaries, hyphenated fallback, env-file creation, and read-only stop/logs/restart behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(docker): reuse the probed Compose binary for wrapper operations The version probe could accept a standalone docker-compose install while COMPOSE_CMD stayed hardcoded to `docker compose`, so preflight passed and start/logs/stop/restart then failed. Keep the selected executable in COMPOSE_BIN (array), refresh COMPOSE_CMD from it in the current shell, and extend the fallback test through an actual stop invocation. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Willem Jiang
|
41b3c17447
|
fix(ci): fix the Agents.md size check test error (#4978) | ||
|
Airene Fang
|
b47c7838a5
|
chore: Extend frontend startup timeout from 120s to 300s. (#4899) | ||
|
Willem Jiang
|
13fe06ee67
|
doc(agent): update the AGENTS.md and ARCHITECTURE.md (#4817)
* doc(agent): update the AGENTS.md and ARCHITECTURE.md * increase the ROOT Agents.md size * Fixed the unit test errors |
||
|
Aari
|
5d520e44a8
|
fix(docker): wait for gateway readiness (#4806)
* fix(docker): verify gateway startup readiness * fix(docker): clarify compose wait requirement |
||
|
Nan Gao
|
c542185a7f
|
feat(extensions): add gateway contribution points and packaged extension management (#4780)
* feat(extensions): add gateway services and routers * feat(extensions): add standalone reference extension * fix(extensions): harden contributed gateway routes * docs(extensions): document gateway contribution points * feat(extensions): add operator CLI for packaged extension management Add `deerflow extensions install/list/enable/disable/remove` plus the root `make extension-*` wrappers, backed by an `ExtensionManager` that owns one transaction over backend/pyproject.toml, backend/uv.lock, the managed source snapshot, the uv environment, and the `plugins:` block in config.yaml. Install accepts a package requirement, a public HTTPS Git URL, or a local directory. Local directories are copied to backend/extensions/sources/ as deployable snapshots rather than editable installs, and the root .dockerignore re-includes that tree so snapshots reach the backend builder. Remote sources are HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock Docker builder cannot reproduce them. Because environment configuration can still resolve a plain package name to a local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed by an audit of the new lock: any local reference the stock image build cannot reproduce rolls back the whole transaction. A config carrying duplicate top-level `plugins:` keys is rejected outright rather than managed against one block while the Gateway reads another. Dependency synchronization now has one lock authority. The `extensions` dependency group joins [tool.uv].default-groups, every startup path syncs the same lock with --locked and launches with --no-sync, and the Docker images move to uv 0.11.1 for the --no-workspace boundary the manager needs. Loader gains `enabled`, `name` and `package` fields so a disabled extension is skipped before resolution and import. Co-authored-by: Codex <codex@openai.com> * fix(extensions): stop the managed plugins rewrite from destroying config Two data-safety defects in the managed `plugins:` block writer. The "next top-level key" boundary was a regex matching only `[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a config may legally carry any top-level key, and a key the pattern cannot recognize did not fail loudly — it read as "no next section", and the rewrite replaced that neighbour and its entire subtree with the managed block. `my.key`, `2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a plain `extension-enable`/`disable`. Both boundaries now come from the YAML parser's node marks, so key shape is irrelevant. The file-final branch never consulted the trailing-comment scan the has-next-key branch used, so any comment below the block was dropped. Since the manager appends `plugins:` at end of file, that is the steady-state shape for most installs: an operator note below the block was destroyed on the next toggle. Separately, every managed install wrote `required: true` while the loader defaults to false. That turned any later load failure — broken wheel, missing native library, deleted snapshot — into a Gateway startup abort recoverable only with shell access. New records are now written `required: false`, with an explicit `install --required` opt-in; adopting an existing hand-written record still preserves the operator's own choice. * fix(extensions): harden the manager transaction and correct its docs Follow-up hardening on the extension package manager. Security posture, which the docs already claimed: - Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and `UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps the interpreter that the entry-point probe then imports and calls, and every later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS validation the HTTPS-only source rule depends on. Neither is an index, proxy, cache or credential-provider setting, so neither was covered by the carve-out. - Recognize run-together and all-caps secret query parameters (`accesstoken`, `ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires on case transitions, so only the separated spellings were caught. Short generic words stay boundary-anchored, so `?keyword=` remains installable. - Validate the config before running any uv command. `uv add`/`uv sync` execute the package's build backend, so a config the manager could never write to must fail before that code runs rather than afterwards through rollback. Transaction integrity: - Run the second dependency-file restore from a `finally`. The recovery sync runs without `--locked` when the checkout had no lock, so uv writes one while resolving; if that sync then failed, the restore was skipped and the operator kept a lock file they never had. A failing recovery sync now also reports the original failure instead of replacing it. - Skip the recovery sync on cancellation. Answering Ctrl-C with a full dependency resolve invites a second interrupt that escapes the handler and strands the checkout mid-transaction; the declarations are already restored and the next locked startup sync reconciles the environment. - Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so contention surfaced as `Permission denied` rather than serializing. - Locate the entry-point probe's JSON payload instead of parsing stdout's first line, so a `sitecustomize`/`.pth` banner cannot roll back a good install. - Warn when the lock records a loopback source. `127.0.0.1` inside the image builder is a different machine, but unlike an environment-driven wheelhouse resolution this is a source the operator typed deliberately, so it is reported rather than rolled back. Private-network indexes are untouched: a builder on that network can reach them. Docs: the blanket claim that failed operations restore the config file was wrong — the conflict branches deliberately preserve a concurrent external edit and leave `remove` deactivated. Document that, the `required: false` default, the config preflight, the interrupt behaviour, and where the plugins-block boundaries come from. * test(gateway): pin the request-path projection agreement `get_request_route_path()` imports the private `starlette._utils.get_route_path` so the auth and CSRF predicates classify the exact string Starlette's router matches on. Its requirement is not "strip root_path correctly" but "return what the dispatcher is matching", so delegating to the router's own implementation keeps the two in lockstep by construction. Keep the private import rather than vendoring a copy: an import that disappears fails loudly at startup, while a stale copy diverges silently at a security boundary. Cover the property directly instead of the mechanism, so the tests survive a future reimplementation: - projection edge cases, including the segment-boundary guard that keeps root_path="/api" from slicing "/apifoo/models" into a string the router would never match - agreement with the router under nested mounts - the two bypasses these predicates exist to prevent: a protected route mounted under the "/health" public prefix must still 401, and a POST mounted under "/api/webhooks" must still require a CSRF token Both are verified to fail when the projection is reverted to `request.url.path` (9/13 red) and when a plausible vendored copy omits the boundary guard (the 2 boundary cases red). Declare starlette as a bounded direct dependency so a bump — which is security-relevant here — shows up in review rather than arriving silently through FastAPI. * ci: pin uv to the version production ships ExtensionManager is not a consumer of uv the build tool -- it is a program whose whole job is driving `uv` as a subprocess, depending on its CLI behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into `[dependency-groups] extensions`) and on the `uv.lock` serialization format. uv is closer to a runtime dependency with a contract than to incidental tooling. backend/Dockerfile pins that binary to 0.11.1, but all eight astral-sh/setup-uv steps installed whatever was latest at run time, so CI exercised the manager against a uv that is not the uv production runs. The sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI stays green because the same uv reads back what it wrote, and the pinned uv in the production image cannot read the committed lock. `uv lock --check` is version-sensitive for the same reason -- it verifies the lock is what *this* uv would produce, and two versions can emit equivalent but non-identical output. Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so the steps share input and caching behavior. Pinning alone drifts apart again on the next bump, so add a constraint test in the style of test_compose_default_bind_host.py: the Dockerfile's UV_IMAGE tag is the single source of truth, and both compose defaults plus every setup-uv step must match it. Verified to fail when a pin drifts, when a step omits `version`, and -- the real scenario -- when the Dockerfile is bumped alone, which lights up the workflows and both compose files at once. * fix(gateway): state the extension route auth limit and abort a failed dev sync Two scoped review follow-ups. README: contributed routers cannot enter the host's reserved public prefixes, which makes every extension endpoint session-authenticated -- there is no way to expose an unauthenticated route. The rejection rule was documented but its consequence was not, so inbound provider webhooks and public status endpoints read as merely undocumented rather than out of scope for this release. docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e` already stopped the script there -- uvicorn was not being started against a stale environment -- but it exited on a bare uv exit code with no indication of what to do. Abort explicitly with the cause and the fix. Tests slice the sync block out of the real script and run it against a stub uv, so they exercise the shipped code rather than a copy of it (/app/backend only exists inside the container). They cover the success path, the retry that recovers, the abort, and the guidance. Verified against the pre-fix script: only the guidance case goes red, confirming the abort itself was already correct. * fix(extensions): point Git SSH shorthand at the HTTPS correction Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git` reached the scheme rules looking like a bare path and was rejected with "local path references are not deployable; pass a local directory so DeerFlow can snapshot it". The operator asked for a remote source, so that guidance points at the wrong fix. Detect the shorthand ahead of the scheme rules and report the public-HTTPS correction instead. The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose `host` reads as a URL scheme and produced the generic HTTPS message. Both spellings now share one message, as does the PEP 508 named form. * docs: keep the root extension summary within its new budget #4799 split the depth out of the module guides and added a size gate; the root file's job is now orientation, and this branch had pushed it 192 bytes past the soft limit. The manager transaction, source rules, and lock discipline are already stated in full in the extensions guide, so the root keeps the one-line orientation and points there instead of restating them. --------- Co-authored-by: Codex <codex@openai.com> |
||
|
Ryker_Feng
|
ccff5f5ce7
|
docs: govern agent guidance size (#4799)
* docs: govern agent guidance size * refactor: split agent guidance by code scope * Clarify virtual path handling in AGENTS.md Updated the translation section to clarify the role of `LocalSandboxProvider` and the handling of virtual paths in the tool layer. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
MasonWight
|
6bb376abfd
|
fix: resolve diagnostic paths from any cwd (#4736)
* fix: resolve diagnostic paths from any cwd * test: cover relative diagnostic script paths |
||
|
ajayr
|
d732b90dc3
|
feat(channels): add Buzz (Nostr) channel connector (#4649)
* feat(channels): add Buzz (Nostr) channel connector Adds a Buzz (https://github.com/block/buzz) channel so DeerFlow can join a Nostr-relay workspace as a member: it answers @mentions in channels, replies to DMs, and streams answers by editing one message in place. * app/channels/buzz_nostr.py — pure NIP-01 helpers: canonical event ids, BIP-340 signing/verification, chat/edit/auth builders, relay frames. * app/channels/buzz.py — BuzzChannel: one NIP-42-authenticated websocket, channel discovery (kind 39000) with one subscription per channel, live membership tracking (44100/44101), per-channel replay watermarks, and replies posted once then edited in place (kind 40003). * app/channels/buzz_run_policy.py — same-thread serialization, mirroring the Feishu precedent. Inbound is gated in order: signature verification, self-drop, /connect bind-and-return, pubkey allowlist, then mention / DM / mention-free / thread-follow. Off by default; needs the new optional `buzz` extra (coincurve, lazily imported), which detect_uv_extras resolves from channels.buzz.enabled the same way it already handles channels.discord. Two relay behaviours drove the design and are worth knowing when reviewing: a global {"kinds":[9]} subscription receives nothing from buzz-relay and a multi-value "#h" filter receives nothing either, so one REQ per channel is required; and a single global `since` cursor skips quiet channels, so watermarks are per channel. Signed-off-by: Ajay R <ajayr@formbuddy.com> * fix(channels): only publish assistant messages from the IM stream `_accumulate_stream_text` decided what streamed `messages-tuple` payloads become displayable text by rejecting ONLY payloads whose `type` contained "tool", so it published everything else. DeerFlow writes hidden model context into the messages channel as ordinary messages -- memory recall and the rewritten user turn as hidden HumanMessages (DynamicContextMiddleware), the `<durable_context_data>` block as another (DurableContextMiddleware) -- and LangGraph fans those state writes out on the messages stream, so they reached every streaming IM channel as the assistant's reply. Proved live on a Buzz relay: the connector published a `<memory>` fact block and, in another run, a verbatim echo of the user's own inbound message. Affects Feishu, Telegram, WeCom and Buzz; worst on Buzz, where each update is an immutable public Nostr event that a corrective edit cannot unpublish. Invert the filter to an allowlist of assistant message types. Two new pure helpers keep it testable: - `_stream_payload_type` resolves the type from both shapes the function already handles: the `model_dump()` shape the gateway emits, and LangChain's `to_json()` constructor shape whose own `type` is the literal "constructor" and whose class name is the tail of the `id` path. - `_is_assistant_stream_type` matches "ai"/"assistant" by PREFIX, not substring -- ordinary words contain "ai" ("chain", "domain"), and a substring test would admit a foreign type name by accident. The bare-`str` branch is removed: an untyped payload cannot be attributed to the assistant, nothing in DeerFlow produces one (serialize_messages_tuple always emits `[message_dict, metadata]`), and a runtime that emitted raw text deltas would emit hidden context the same way. Per-message-id buffering and merging are unchanged. Tests pin both directions, including multi-chunk merging across one message id, so the allowlist cannot silently kill streaming, plus an end-to-end `_handle_streaming_chat` test asserting the live payload never reaches an outbound message. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Ajay R <ajayr@formbuddy.com> * chore(helm): bump config_version to 33 in chart values and README config.example.yaml moved to 33 for the buzz channel block; the chart's embedded config example and its README copy track it (config_version only drives the outdated-config warning, per scripts/check_config_version.sh). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ajay R <ajayr@formbuddy.com> --------- Signed-off-by: Ajay R <ajayr@formbuddy.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
天魔
|
276481371e
|
fix(docker): set DEER_FLOW_ROOT for log commands (#4658)
* fix(docker): set DEER_FLOW_ROOT for log commands * fix(docker): set root before restart * docs(docker): clarify log command |
||
|
Vanzeren
|
095092418c
|
fix(gateway):unify thread id validation (#4589)
* fix(gateway): unify thread ID validation at the API boundary
Thread ID entry points accepted arbitrary strings while downstream
consumers (filesystem paths, Kubernetes Provisioner, JSONL event store)
each enforced different character restrictions, so invalid IDs were
persisted first and only failed later during sandbox/workspace init.
Centralize validation in deerflow.utils.thread_id (pattern
^[A-Za-z0-9_-]{1,64}$): validate at routers, RunCreateRequest,
scheduler dispatch, paths.py, JSONL store, embedded client, and align
the Provisioner pattern (pinned by a parity test). UUIDs are still
generated only when no ID is supplied; caller-supplied opaque IDs stay
supported.
Deliberate exceptions: DELETE /threads/{id} keeps str as the legacy
cleanup escape hatch (filesystem cleanup guarded), read-only
client.get_thread stays unvalidated, and scheduler rows with legacy
invalid IDs record a failed dispatch instead of raising out of the
poll loop.
* docs: document canonical thread ID contract
README: caller-supplied thread IDs need not be UUIDs; the canonical
pattern and per-endpoint behavior. AGENTS.md: the shared
deerflow.utils.thread_id contract, its enforcement boundaries, and the
legacy-ID escape hatches.
* fix(gateway): close thread ID validation gaps at remaining entry points
Follow-up to the canonical thread ID contract: a full audit found the
uniform-422 coverage only reached about half of the thread_id surfaces.
- routers: 18 routes still took a bare thread_id: str — 13 in
thread_runs.py (including the five messages/events/workspace-changes
reads that returned 500 on the JSONL event store vs 404/empty on the
DB store), 4 read routes in threads.py, and the suggestions route
flagged in review. DELETE /api/threads/{id} keeps str as the declared
legacy-cleanup escape hatch.
- client: upload_files/delete_upload/list_uploads/get_artifact now
validate up front, fulfilling the RFC's 'all mutating entry points'
clause (get_thread stays unvalidated as the declared legacy read path).
- tui: the /resume literal-ref fallback validates against the canonical
contract and reports a descriptive error instead of failing deep in
the client.
- scripts/support_bundle.py: replace the drifted dot-allowing pattern
with a byte-identical copy of THREAD_ID_PATTERN (kept local so the
script still runs with a broken venv).
* test(gateway): guard the canonical thread ID contract against regressions
- test_thread_id_route_contract.py: static AST sweep asserting every
route handler with a thread_id parameter annotates ThreadId
(whitelist: the DELETE escape hatch), plus a runtime sweep hitting
all 44 thread_id routes with a non-canonical ID and asserting a 422
that names thread_id, plus a websocket upgrade-rejection case.
- test_thread_id_validation.py: client entry-point validation,
support_bundle pattern parity, and TUI literal-ref fallback tests.
- Align two tests that encoded the old contract (dotted IDs).
|
||
|
Nan Gao
|
2a143dced6
|
fix(docker): bind the published entry port to loopback by default (#4618)
README documents DeerFlow as deployed by default "in a local trusted
environment (accessible only via the 127.0.0.1 loopback interface)", but both
compose files published nginx as `"${PORT:-2026}:2026"`, which Docker binds to
0.0.0.0 and [::]. The shipped artifact did not match its own documented
default, so running it on a LAN or cloud host produced a wider surface than
the docs implied without the operator changing anything -- and the agent can
execute commands.
Publish as `"${BIND_HOST:-127.0.0.1}:${PORT:-2026}:2026"` in both compose
files, so the default matches the documented model while operators who front
the stack with their own TLS/auth can still widen it via BIND_HOST. The
Gateway keeps binding 0.0.0.0:8001 inside the container (nginx reaches it over
the compose network) and its port stays unpublished, so the published nginx
port is the entire external surface.
BREAKING CHANGE: a deployment that relied on the previous 0.0.0.0 default
becomes unreachable from other hosts after this upgrade. Set BIND_HOST=0.0.0.0
in .env to restore it, after putting authentication in front and completing
first-run setup.
Also:
- .env.example documents BIND_HOST and PORT with the reasoning.
- deploy.sh reports the address the stack actually bound and, when it is not
loopback, tells the operator to complete first-run setup immediately. It
reads BIND_HOST/PORT from .env via a new read_dotenv_value helper following
compose precedence; the shell does not source .env, so reading the
environment alone would have reported "loopback only" for a stack .env had
exposed. The pre-existing ${PORT} summary line had the same defect and is
fixed with it.
- test_compose_default_bind_host.py pins the loopback default, that BIND_HOST
stays overridable, and that no service in either compose file publishes a
port without an explicit bind address, so a later addition cannot drift back
to 0.0.0.0 unnoticed.
|
||
|
jinhaosong-source
|
d0957409f1
|
feat(wizard): add OrcaRouter as an LLM provider (#4598)
OrcaRouter is an OpenAI-compatible routing gateway. Mirror the existing OpenRouter entry in the setup wizard's LLM_PROVIDERS: reuse langchain_openai:ChatOpenAI pointed at api.orcarouter.ai/v1 with env var ORCAROUTER_API_KEY. Default model pins a tool-capable model; orcarouter/auto is also selectable. Disclosure: I'm an engineer on the OrcaRouter team. Co-authored-by: jinhaosong-source <jinhaosong@myflashcloud.com> |
||
|
ShitK
|
4e44938551
|
fix: align pnpm consumers with Corepack fallback (#4405)
* fix: align pnpm consumers with Corepack fallback * fix: run pnpm helper from frontend workspace * fix: preserve Corepack resolution hint |
||
|
nonoge
|
183280ebfc
|
fix browser extra detection for indentless YAML (#4367) | ||
|
Ryker_Feng
|
fa496c0c8d
|
feat(browser): add agentic browser control (#4187)
* feat(browser): add agentic browser control
* fix(frontend): format browser view changes
* fix(browser): keep browser optional and isolate sidecar layout
* fix(browser): address PR review security and IME findings
- Nginx: add a browser-stream WebSocket location before the generic
/api/threads regex so Live upgrades instead of downgrading to HTTP
(both nginx.conf and nginx.local.conf).
- Ownership: require an existing owned thread for the WS stream and REST
navigate, and tear down the browser session on thread deletion so a
later caller cannot reuse a retained page/cookies by guessing the id.
- SSRF: enforce the URL policy at the browser request boundary via a
context-level route guard covering redirects, popups, iframes, and
subresources (skipped for CDP-attached Chrome).
- IME: skip key forwarding while a composition is active so confirming a
CJK candidate with Enter no longer submits the remote page form.
Adds regression tests for the request guard, session teardown on delete,
and the composing-Enter key decision.
* fix(frontend): smooth streaming in long tool threads
* Revert "fix(frontend): smooth streaming in long tool threads"
This reverts commit f0462516eabe77f138d4027ea1c714fb226683cf.
* fix(browser): address review security and lifecycle findings
- Reject cross-origin WebSocket upgrades on the live browser stream
(Origin allow-list reuse of CORS/same-origin helpers) to close a
WS-CSRF hole, and fail closed when the ownership store is absent.
- Warn when a CDP-attached session runs with the SSRF request guard
off, and drop the unreachable CDP screencast teardown dead code.
- Read browser session launch config from a single canonical source
(browser_navigate) so it is deterministic regardless of call order.
- Bound per-thread Chromium accumulation with idle-timeout eviction
and an LRU max-sessions cap.
- Reset the Live reconnect counter on a successful open so the stream
can't permanently stall after the cumulative attempt cap.
* fix(frontend): reduce long tool thread render stalls
Reuse stable historical message groups during streaming, defer heavy Markdown and browser previews, and lazy-decode message images.
* fix(browser): keep live control responsive during continuous input
Why: Manual browser control felt laggy — a physical click ran the remote
Playwright click three times and each non-move input synchronously awaited a
JPEG screenshot, so events queued behind capture (queue wait up to ~237ms).
The first async attempt used a trailing-edge debounce, which froze the visible
page until a wheel/keyboard gesture stopped ("scroll finishes, then it jumps").
What:
- Frontend forwards one `click` per physical click instead of also emitting
`down`/`up`, so the remote page is not clicked twice per gesture.
- Backend detaches live-frame capture from input dispatch: non-move actions
start a rate-limited background refresh loop (leading frame + bounded cadence)
that keeps emitting frames while input continues and never blocks dispatch.
- Add regression tests: input dispatch no longer awaits the screenshot, rapid
inputs coalesce, and continuous input keeps refreshing before it stops.
Scenarios: Verified in the live Browser panel — a single click completes in
~57ms (was blocked behind a 171ms capture), and a 1.14s sustained wheel gesture
renders ~7 frames throughout the scroll instead of one frame after it ends.
* fix(browser): harden worker and session lifecycle
* fix(browser): address latest review feedback
* fix(frontend): preserve optimistic new-chat message
* test(e2e): preserve mocked message run ids
* fix(browser): address capability review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Aari
|
5eb59cb130
|
fix(sandbox): stop multi-worker orphan reconcile from killing peer sandboxes (#4221)
* fix(sandbox): stop multi-worker orphan reconcile from killing peer sandboxes Docker sandboxes are shared across gateway workers, but each worker kept its own in-memory warm pool. Startup reconciliation adopted every running container, so a peer idle reaper could destroy sandboxes another worker still owned and tool calls hit 502 / Connection refused. Add file-based ownership leases under sandbox-leases/, only adopt true orphans, refuse idle/replica/shutdown destroy while a foreign lease is live, and renew the lease on create/get/release/reclaim. Fixes #4206 * fix(sandbox): close lease fail-open, hot-path IO, and check→destroy race Address review of the multi-worker orphan lease (#4206): - read_lease returns None only for a genuinely-absent lease and raises (CorruptLeaseError/OSError) when a lease is unreadable or corrupt, so the ownership check fails closed instead of mistaking an unprovable peer lease for a free container. clear_lease still removes a stuck/corrupt file. - get() no longer renews the lease (blocking mkdir/fsync/os.replace on the event loop path used by ensure_sandbox_initialized_async); active leases are renewed off the event loop from the idle checker (_renew_active_leases). - The ownership check and container stop run under a per-sandbox flock guard (lease_ownership_guard); every lease write takes the same guard so a peer's touch cannot interleave with a destroy. Same-host multi-worker scope, not a multi-pod distributed lock. Also fixes the ruff format lint on the branch. Adds regression tests: corrupt and unreadable lease fail closed, a tests/blocking_io anchor keeping get() non-blocking on the event loop, and a peer-touch/destroy interleave test. * fix(sandbox): share container ownership across gateway instances Rework of the #4206 fix per review: ownership state is shared through a third-party service instead of being maintained per gateway instance, following the stream_bridge precedent (sandbox.ownership.type: memory | redis). The file lease and its same-host flock guard are deleted, not ported — they only covered workers on one host, while the deployment that hits #4206 is a load-balanced multi-instance gateway. A lease answers "who reaps this container", not "who may use it". Containers are deterministic per (user, thread), so consecutive turns legitimately land on different instances: take() transfers ownership on acquire, while claim() gates every adopt/reap path. Leases carry a state — own: or del: — so a takeover is refused against a teardown in progress. Without it an unconditional take() would overwrite a destroyer's claim and the peer's container stop would land on a sandbox the new owner had already handed to an agent. renew() distinguishes a lapsed lease from one a peer took; only the latter drops the sandbox. Collapsing them meant a Redis restart evicted every in-flight sandbox on every instance at once. Renewal runs on its own thread with a TTL derived from its interval, never from idle_timeout: renewal used to ride the idle checker, which does not start at idle_timeout: 0, so leases silently lapsed on a supported config. Ownership establishment is fail-closed: a sandbox whose ownership cannot be published is never handed out, and a just-created container is destroyed rather than leaked as an adoptable orphan. Every destroy path claims before untracking. The memory store is single-instance only and says so; the resolver reads app_config.stream_bridge and the env var in the bridge's own order, so deployments already using Redis get a redis ownership store without extra config. * fix(sandbox): wait out a recovery grace before adopting a keyless container An absent ownership lease meant two opposite things on two paths. Renewal reads it as LAPSED and re-establishes it: nobody took the lease, so the container is still ours. Reconciliation read the same absent key as "orphan" and adopted on sight. After the store loses its keys (a Redis restart without persistence, or eviction under maxmemory) every owner is alive and merely pre-renewal-tick. Whichever instance reconciled first therefore adopted every live container; each real owner's next renewal reported LOST and dropped a sandbox it was serving mid-turn, leaving it for the adopter to idle-destroy — #4206 through the back door, in the very case the LAPSED handling was added to make safe. Not limited to startup: an already-running instance hits the same window from the idle checker's periodic reconcile. _adoptable_after_grace requires an untracked container to be seen unowned across a full lease TTL before it can be adopted. That rebuilds the delay the state loss erased: a live owner republishes within one renewal interval, shorter than the TTL by construction, while a crashed owner never does, so its containers are still adopted one grace later rather than leaking. A republished lease resets the grace; a pausing-only timer would still expire over a live owner's lease. The peek is read-only — the atomic claim still gates adoption. The grace is skipped when the store cannot coordinate across processes: no peer can hold a lease such a store would show us, so single-instance deployments keep instant orphan cleanup, and a grace could not help a multi-worker gateway on memory anyway. * fix(sandbox): hold the teardown lease for as long as the container stop runs claim(..., for_destroy=True) wrote the del: marker with the ordinary lease TTL and nothing refreshed it. renew() extends only own: and deliberately reports a teardown as LOST, and the destroy paths drop the sandbox from the maps the renewal loop iterates — so a container stop that outlived the TTL let the marker lapse, a peer's take() succeeded against the still-running container, and the stop then landed on the turn that had just been handed it. That is the exact window the del: state exists to close, reopened by its own expiry. The two lease states alone never made the per-sandbox flock redundant, as I claimed when deleting it: a held lock cannot expire, a lease can. The exclusion has to be held deliberately rather than assumed to outlast the work it guards. _held_teardown_lease wraps both _backend.destroy() call sites and re-claims the marker every renewal_interval_seconds until the stop returns. No store change is needed: claim(for_destroy=True) already refreshes an existing del: marker on both backends. Reachable without an abnormal backend. The schema bounds only renewal_interval_seconds (> 0) and ttl_multiplier (>= 2), so a legal config puts the TTL below a normal container stop; and LocalContainerBackend._stop_container passes no timeout to subprocess.run, so a wedged daemon blocks unbounded even at the default 120s TTL. The TTL stays finite on purpose: the heartbeat dies with the process, so a destroyer that crashes mid-stop still releases the container one TTL later instead of marking it undestroyable forever. * fix(sandbox): hold the teardown lease on every del: stop, and pin the claims that had no test 90936b49 said `_held_teardown_lease` wrapped "both" `_backend.destroy()` call sites. There are three. `_drop_unhealthy_sandbox` marks `del:` and then blocks on the same unbounded stop, and it untracks *before* claiming, so `_renew_owned_leases` cannot see the id either — nothing refreshed the marker. Reproduced against a real redis: the peer's `take()` succeeds 1.0s into a 2.5s stop. Same window, third path. That miss came from the habit the rest of this commit addresses: a property asserted in prose, with no test that could falsify it. Auditing every load-bearing claim in this feature — AGENTS.md, the store docstrings, the provider's design comments — against the test that would go red turned up several more, each verified by mutating the code and watching the suite stay green. Tests that could not fail: - `test_reconcile_fails_closed_when_ownership_unknown` reached the grace gate, not the claim. A bare MagicMock answers `owner()` with a truthy mock, so the container read as peer-owned and deferred; `claim()` was never called. It stayed green with `_claim_ownership` failing open. Adding the grace ahead of the claim is what hollowed it out — inserting a gate can silently disarm the tests for the gate behind it. - `test_adoption_grace_restarts_when_a_live_owner_republishes` never distinguished reset from pause. Those diverge only on a *second* lapse, which it never drove, so it passed with the reset deleted. Claims with no test at all, each now pinned (mutation → red, per test): - `destroy()`, `_evict_oldest_warm`, `_reclaim_warm_pool_sandbox`, `_register_created_sandbox` and `shutdown()`'s warm loop were each the one untested sibling of an "every path does X" enumeration. `shutdown()` was never driven with a non-empty warm pool, so a loop bypassing the ownership claim — stopping a live peer's container on our exit — went unnoticed. - Renewal's unknown-is-not-lost rule, the single deliberate exception to fail-closed. Inverting it drops every active and warm sandbox on every instance the moment the store blinks. - Both hops of the stream-bridge redis inference. Deleting either left the suite green while every config.yaml-native multi-instance deployment silently fell back to memory — #4206 reopened on exactly the deployments the inference exists for. Claims narrowed instead, because they promised more than the code delivers: - "run against both backends ... cannot drift" — CI provisions no redis, so the merge gate runs the memory tier only and the Lua never executes there. - "Every destroy path claims before untracking" — `_drop_unhealthy_sandbox` untracks first, deliberately, under its `expected_info` TOCTOU guard. - "Atomic: concurrent claims from different instances cannot both succeed" — true via Lua on redis, vacuous on the single-instance memory store, and pinned by neither, since the contract suite drives sequential calls. A concurrency test against the memory store would make the claim look covered while the mechanism that carries it still never runs in CI. * fix(sandbox): release the teardown marker when a destroy() stop fails The three `del:`-marked stop paths disagreed on failure. `_destroy_warm_entry` releases on both outcomes and says why: the stop failed, so the container is probably still up, and a marker left behind refuses its own thread's `take()` until the TTL lapses. `_drop_unhealthy_sandbox` does the same. `destroy()` had no such guard — a raising backend propagated straight past `_release_ownership`, and the thread could not re-acquire for a full TTL. Fails safe rather than fatal: a stuck marker stops peers from touching the container, it is not the cross-instance kill. But the paths must agree, and this one is the odd one out. Release, then re-raise. Swallowing would be the easier symmetry with `_destroy_warm_entry`'s `return False`, but `destroy()` has no failure return and `shutdown()` logs per sandbox off the exception, so swallowing would silently narrow what callers can see. Found by comparing the three paths after @fancyboi999 asked for release to be handled "consistently with the other destroy paths" on the unhealthy path — which 0d2377b2 already does. This is the sibling that wasn't. * fix(deploy): bump chart config_version to 27 for sandbox.ownership config.example.yaml went to 27 with the new sandbox.ownership section, but the chart embeds its own copy and stayed at 26, so validate-chart failed. A bare bump: the chart already sets stream_bridge.type=redis, which is what resolve_ownership_config infers a redis ownership store from, so no field change is needed. * fix(sandbox): release the teardown lease from its heartbeat, not the caller `_held_teardown_lease` joined its heartbeat only briefly and the caller cleared the `del:` marker right after the stop. A refresh `claim` still in flight (`RedisOwnershipStore` had no socket timeout, so a round trip could block) could land *after* that release and rewrite `del:` on a container whose stop had already completed — refusing a fresh `take()` (or rolling back a fresh create) until the TTL. Move the release into the heartbeat's own `finally`, after its loop stops, so no refresh can run after it. The three destroy paths no longer release after the `with` (`destroy()`'s no-container branch still does, since no lease was held there). Bound every store round trip with a socket timeout so the in-flight refresh — and thus the deferred release — stays finite, and broaden the heartbeat's `except` so an unexpected error cannot strand the marker during a long stop. Also fold in the review follow-ups: stop re-resolving an already-resolved ownership config in the factory, document the Redis-outage-vs-TTL boundary in config.example.yaml, and add a tests/blocking_io anchor pinning that `release()`'s store round trip stays off the event loop. * fix(sandbox): refuse a non-destroy claim that would unwind our own teardown `claim(for_destroy=False)` against our own `del:` lease fell through and overwrote it with `own:`, cancelling a teardown that was already in flight. The container stop cannot be recalled, so downgrading the marker would let a `take()` hand out a container that is about to die -- #4206, self-inflicted. No caller does this today: the two non-destroy callers run against an absent key (the LAPSED re-claim) or an unowned one (post-grace reconcile). The contract has to forbid it rather than rely on that staying true. Fixed in both backends. The redis rule lives in Lua and the memory rule in Python, so fixing one only would let them drift silently -- and the shared contract suite is what is supposed to catch that drift, so it now covers this. Also adds a contention test for `claim`. The suite drove sequential calls only, so it pinned the exclusion predicate but not the atomicity that predicate depends on; eight instances now race for one container and exactly one must win. * fix(sandbox): bound the container stop so it cannot outlive its teardown lease `_stop_container` passed no `timeout` to `subprocess.run`, so a wedged container runtime blocks it forever. The `del:` marker is what keeps a peer from re-acquiring the container while the stop runs, but a marker is a lease and a lease can lapse: a store outage longer than the TTL frees it, a peer's `take()` succeeds against the still-running container, and the stop then lands on the turn that was just handed it -- the exact #4206 failure. The teardown heartbeat already covers the case where the store stays reachable. This bounds the worst case independently of the ownership layer, which is the point: it holds even when the ownership layer is the thing that failed. A timeout is not swallowed like a `CalledProcessError`. That error means the runtime answered "I could not stop it"; a timeout means we do not know, and the container is probably still running -- returning normally would let `_destroy_warm_entry` report a clean stop and drop the warm entry, leaking a running container nothing tracks. * fix(sandbox): exclude this instance's own reapers from its acquire path An ownership lease excludes peers and nothing else. `claim()` and `take()` both succeed against our own `own:` lease by design -- that is what lets a destroy path claim what it already owns -- so `del:` says nothing to this process's other threads. Meanwhile every reaper decides outside `_lock`, because a store round trip must not be held under the lock that guards every acquire. So each reaper acts on a decision its own acquire path may already have invalidated, and the store cannot see the difference. Six paths end in an irreversible act (a container stop, or closing a host-side client) on a decision made outside the lock. All six reproduce: _evict_oldest_warm re-checks warm membership, then releases the lock _reap_expired_warm no re-check at all _cleanup_idle_sandboxes re-verifies idle, then releases the lock _renew_owned_leases acts on a stale renew() -> LOST release() same staleness on its own refresh _drop_unhealthy_sandbox untracks before claiming, opening discovery Both warm reapers are a regression from the deferred pop this branch introduced: `WarmPoolLifecycleMixin` popped under the lock, so a reclaim's membership check failed and the race could not occur. Deferring the pop is still right (popping first loses the container on a refused claim), so the exclusion has to be made explicit instead. The idle path is pre-existing in shape, but this branch widened it from a few instructions to a network round trip by claiming ownership before untracking. Two guards, because the two directions want opposite answers: Reaping -- nothing may promote it. The reaper reserves the id, and every promote path refuses a reserved id exactly as it refuses a peer's `del:` (drop and cold-start). The "is this still reapable?" test travels with the reservation as a predicate and runs in the same critical section, because checking first and reserving second is the window, not a narrower version of it. Forgetting -- the peer legitimately wins, so the promote is what to detect. `_publish_ownership` bumps a per-id acquire epoch; the callers that decide from a store round trip snapshot it first, and the pop is skipped if it moved. Object identity cannot substitute: the reuse path re-publishes ownership while handing out the same tracked `AioSandbox`, so an identity check sees nothing and the pop closes a client mid-turn. `still_reapable` is required rather than defaulting to unconditional -- the safe default is the one that makes a new call site think about it. That diverges from the mixin hook, which is safe because this provider overrides both mixin callers, and loud rather than silent if those are ever dropped. Also closes a client leak on the discover path: "nothing to roll back" was true of the container but not of the HTTP client constructed before the publish, which the sibling create path already closes. The shared-store test view rebound `owner_id` outside the store's lock, so a concurrent claim could execute under the wrong id and read its own lease as a peer's. Serialized, so the heartbeat-hold tests stop flaking. * fix(sandbox): mark acquire intent before the ownership round trip A guard must become visible no later than the transition it guards. The acquire epoch cannot manage that for `take()`: the takeover is durable before `take()` returns -- redis has committed the SET while the reply is still in flight -- and the epoch can only be written afterwards. In that interval the store already says the container is ours while the epoch still reads as it did when a renewal decided `LOST`, so the stale forget walks through, drops the maps and closes the client the acquire is about to hand back. Acquire then returns an id the provider no longer tracks and `get()` answers `None` for the rest of the turn. `_publish_ownership` now publishes an intent mark under `_lock` before the round trip; the epoch keeps covering the other half, "an acquire completed since you decided". `_forget_lost_sandbox` honours the intent mark unconditionally rather than only when an epoch is supplied -- today's epoch-less callers cannot reach the window, but "no epoch" reading as "no guard" is how the next caller of a dangerous primitive gets written. The same invariant had four more instances, all reproduced: reuse returns a decision the forget already invalidated -- before the mark is set a `LOST` is both current and correct, so the forget legitimately runs and the entry reuse decided to hand out is gone. Re-check after publishing and fall through to discovery instead. reclaim installs an entry a reaper reserved after its check -- the warm entry is still visible during the stop, and the reaper's claim succeeds because reclaim's own take() just made the lease ours. Re-check likewise. the reservation was released before the entry was removed -- the pop belonged to the caller, leaving a gap where the container is stopped, the entry is still in `_warm_pool`, and nothing marks it. `_destroy_warm_entry` removes it itself, inside the reservation; the pop stays deferred relative to the stop, just not to the reservation. reconcile adopts a container this instance is tearing down -- adoption is a promote and needs the same reservation check as the others. Neither existing guard excludes it: the claim succeeds because the lease is ours, and on `memory` the recovery grace is skipped outright. The pre-round-trip checks in reuse and reclaim are kept as early-outs, since they skip a health check and a store round trip on a doomed entry, and are pinned to that job rather than to a correctness role they no longer hold. The teardown reservation predicate runs under `_lock`, so it must not touch the lock. Documented rather than engineered around: making the lock reentrant to tolerate it would trade a loud hang for a quiet class of re-entrancy bugs across the rest of the provider. * fix(sandbox): honor local teardown after ownership publish * fix(sandbox): clear a stale warm entry when an id becomes active Active and warm are exclusive states, and the two register paths were the only place that could hold both: they inserted into `_sandboxes` without popping `_warm_pool`, so one container ended up with two reapers. `_reap_expired_warm` judges an entry by its warm timestamp and never consults `_last_activity`, so it stops a container an agent is actively using while `_sandboxes` still hands out its client. Reachable because `_reconcile_orphans` adopts an untracked-but-running container into the warm pool inside the register's publish -> track window, and on the `memory` store it adopts on sight: `_adoptable_after_grace` short-circuits when `supports_cross_process` is False, so an id carrying this process's own lease reads as adoptable. That window is new to this branch -- on main the track was a single locked insert with nothing before it. Both register paths now pop the warm entry inside the same locked section that installs the active one. * fix(sandbox): harden ownership renewal teardown --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Daoyuan Li
|
283cea567e
|
fix(scripts): broaden support bundle secret-key redaction denylist (#4242)
* fix(scripts): broaden support bundle secret-key redaction denylist SECRET_KEY_RE only matched a fixed keyword allowlist, so a secret stored under an unanticipated key name inside an open-ended config dict (e.g. guardrails.provider.config, an arbitrary provider-kwargs dict) was emitted verbatim into config-summary.json even though manifest.json claims redacted_secret_fields=true. This gap was flagged on PR #3886's review before merge but not fully addressed. Broaden the key-name match to mirror env_policy.py's wildcard denylist (*KEY*/*SECRET*/*TOKEN*/*PASS*/*CREDENTIAL*/*DSN*) already used for sandbox env-scrubbing, plus its no-flag credential exact names (GH_PAT/GITHUB_PAT/ REDIS_AUTH/REDISCLI_AUTH/PGSERVICEFILE). The new bare key/pass/dsn alternatives are boundary-guarded so they match only their own delimited token, not an unrelated word that starts with the same letters (routing "keywords", guardrails "passport"). * fix(scripts): stop the pass token boundary from missing passphrase/passcode SECRET_KEY_RE's bare "pass" alternative, (?<![a-zA-Z])pass(?![a-zA-Z]), excludes any key where "pass" is followed by another letter. That correctly keeps "passport" out of the redaction set, but it also excludes genuine secret-bearing key names like "passphrase" and "passcode" -- both of which env_policy.py's *PASS* substring denylist does catch, so a secret stored under either name in an open-ended config dict (e.g. guardrails.provider.config) would still leak into config-summary.json. Narrow the lookahead to only exclude a trailing "port" -- pass(?!port) -- so passphrase/passcode/pass/db_pass all match while passport stays excluded; compass/bypass stay excluded via the existing leading-letter lookbehind, independent of the lookahead. Added a regression test covering both the newly-caught names and the still-excluded ones in one place. Reverting to the old lookahead reproduces the exact leak (passphrase left unredacted); with the fix, tests/test_support_bundle.py (30 tests) is green, and ruff check/format are clean. |
||
|
Zheng Feng
|
d57f695769
|
fix(helm): default sandbox Services to ClusterIP (#3929) (#4190)
* fix(helm): default sandbox Services to ClusterIP (#3929) The K8s sandbox provisioner supports both NodePort and ClusterIP via SANDBOX_SERVICE_TYPE (added in #4016), but the Helm chart never set it, so real-cluster installs inherited the NodePort default. That bound the code-execution sandbox on every node's interfaces - including externally reachable ones on GKE/EKS/AKS - and pinned every sandbox URL to one node IP (SPOF on node reboot/drain/ephemeral-IP). Default the chart to ClusterIP: the provisioner returns a cluster-DNS URL (http://sandbox-<id>-svc.<ns>.svc.cluster.local:8080) so the gateway-> sandbox hop stays inside the cluster network - no node IP, no 30xxx port, no external exposure. The chart always runs the gateway in-cluster, so ClusterIP is always correct there. NodePort remains an opt-in (provisioner.sandboxServiceType: NodePort + nodeHost) for the Docker-Compose/hybrid path where the gateway is not in K8s and cannot resolve .svc.cluster.local; the provisioner code default stays NodePort for that path. - values.yaml: add provisioner.sandboxServiceType ("ClusterIP") - provisioner-deployment.yaml: emit SANDBOX_SERVICE_TYPE; gate the NODE_HOST block on NodePort mode (default "ClusterIP" for upgrade safety) - NOTES.txt + README.md: document ClusterIP default + NodePort opt-in No change to docker/provisioner/app.py (already mode-aware since #4016) or RBAC (services verbs already cover ClusterIP). * test(helm): assert sandbox Service-type gating + CHANGELOG the default flip (#3929) Address review on #4190: - Add scripts/check_chart_sandbox_service.sh: renders the chart for the default (ClusterIP, no NODE_HOST), the NodePort opt-in (both emitted), and NodePort+nodeHost (literal value, not downward API). Locks in the #3929 gating so a regression (e.g. re-adding an unconditional NODE_HOST, or dropping the `default "ClusterIP"` upgrade-safety fallback) fails CI. Wired into .github/workflows/chart.yaml validate-chart job. (#2) - CHANGELOG [Unreleased] -> Changed: note the NodePort->ClusterIP default flip on upgrade + the `sandboxServiceType: NodePort` opt-back-in. (#4) No chart template changes (the gating itself landed in the first commit). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Daoyuan Li
|
656f6b364c
|
fix(skills): recognize fully deleted public skill packages in review CI (#4169)
select_skill_packages() resolved every changed non-SKILL.md path to its owning package via an unconditional depth-3 fallback, then queued that path for review. When a PR deletes an entire public skill package (not just SKILL.md, but its scripts/assets/etc. too), the fallback still returned the package directory even though it no longer exists on disk post-deletion. The review CLI then reported a false structure.missing-skill-md blocker for a path that isn't there, failing CI on a routine, correct package removal. Skip a resolved package only when every changed file under it was a deletion and the package directory itself is gone from disk - i.e. the whole package was intentionally removed. A package left in a broken/partial state (e.g. SKILL.md deleted while sibling files remain) still resolves to an existing directory, so it is unaffected and continues to be queued and flagged. |
||
|
pclin
|
fabadae416
|
add Volcengine Coding Plan to quikly setup (#4141)
* add Volcengine Coding Plan to quikly setup * modify model list |