mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-02 00:20:40 +00:00
2584 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
ChengFa
|
78540ac0e4
|
fix(scheduler): preserve unchanged once schedules near execution (#6686)
* fix(scheduler): preserve unchanged once schedules near execution * docs: keep schedule guidance within the size limit * style(scheduler): use English for the once-edit comment * docs(scheduler): keep guidance within directory budgets --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Jason Cat
|
ca81ace2dc
|
fix(acp): resolve an absolute Windows agent command before spawning (#6656)
* fix(acp): resolve an absolute Windows agent command before spawning A command configured as an absolute path without an extension -- npm's global shim layout, e.g. C:\npm\mcode beside mcode.cmd -- was handed to the ACP SDK unchanged and always failed: create_subprocess_exec applies no PATHEXT, so it ran npm's extensionless shell script and died with WinError 193 (or raised FileNotFoundError when only the .cmd exists). The separator guard from #6598 was too broad: only relative paths need to stay with the spawn, because their meaning is owned by the ACP-workspace cwd, while an absolute path does not depend on the cwd. Absolute paths now go through the same shutil.which lookup as bare names: the lookup searches the command's own directory and probes PATHEXT, and the configured value is kept when nothing matches so the not-found remediation still fires. Relative paths (bin/agent, ./bin/agent) still go to the spawn unchanged. Docs: README.md and backend/packages/harness/deerflow/tools/AGENTS.md now describe the bare/absolute vs relative split. AI disclosure: this change was drafted with AI assistance at my request, under my direction and review. * fix(acp): keep already-absolute lookup results unchanged Review feedback on #6656 (willem-bd): os.path.abspath collapsed a `..` that follows a symlink, so an absolute POSIX command like `/srv/current/../shared/agent` (with `current -> releases/v1`) was rewritten to `/srv/shared/agent` -- a path that either does not exist or names a different launcher. The anchoring to the Gateway cwd exists only for relative PATH lookup results, where shutil.which cannot know the cwd the spawn will use; an already-absolute result needs no rewrite. Regressions: a platform-independent stub-lookup test pinning the literal parent-segment path, and a POSIX-only test that builds the symlink layout and asserts the configured command is returned unchanged. |
||
|
Totoro
|
5e67d41c47
|
fix(goal): validate goal text and preserve the current goal on message replay (#6685)
* fix(goal): validate goal text and preserve current goal on message replay * fix(goal): reject malformed continuation counters before admission * fix(goal): reject external reducer overwrite markers --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> |
||
|
ordinary being
|
a7dd4849eb
|
fix(runtime): mark cancel takeovers with the orphan-recovery stop reason (#6674)
* fix(runtime): mark cancel takeovers with the orphan-recovery stop reason A cross-worker cancel that takes over an expired-lease run writes error but left stop_reason unset. `orphan_recovered` is the only signal that lets a store-only SSE/`wait` consumer synthesize END after a heartbeat, because the producer is known to be gone. Without it, a subscriber that outlives the crashed owner waits forever for an END no surviving worker publishes; periodic reconciliation ignores the row because it is already terminal. Mark the takeover like periodic orphan recovery: claim_for_takeover already accepts the stop reason and the reconcile path already passes it. The new tests pin the takeover row's marker and prove a real takeover wakes a waiting consumer; three existing takeover test doubles now forward the store argument. * docs(gateway): keep guidance within the 48KB budget The takeover stop-reason sentence pushed backend/app/gateway/AGENTS.md over its 48KB hard limit, failing agent-guidance and the backend guidance-budget test. Trim the sentence; the rationale stays in the manager.py comment. * fix(gateway): complete streams after scheduler orphan recovery --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
29a958100e
|
fix(sandbox): stop resolving sandbox and ACP env values a second time (#6673)
* fix(sandbox): stop resolving sandbox and ACP env values a second time AppConfig.resolve_env_variables already replaces every $VAR when config.yaml loads. The AIO, E2B and OpenSandbox providers and the ACP tool then resolved the result again, so a secret that itself starts with $ (a bcrypt hash, a generated password) was looked up as a variable name and reached the sandbox or agent as "" -- or as an unrelated host variable's value when that name existed. Pass the loaded values through unchanged, as BoxLite and the local provider already do. The removed helpers' str() coercion was dead: both fields are typed dict[str, str] and reject non-strings at load. * docs(changelog): reference #6673 * docs(config): pin when sandbox and ACP env values are resolved ACPAgentConfig.env and the SandboxConfig docstring said $ values are resolved from the host environment without saying when, which reads as "at spawn time" and invites a consumer-side second pass again. Both now say it happens when config is loaded, matching SandboxConfig.environment. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Jason Cat
|
37ff61e87b
|
fix(tests): make oversized-payload tests pass on Windows hosts (#6654)
test_skillscan_env_launcher.py and test_channels.py put payloads of up to 52 KB directly in parametrize values, so pytest's generated item ids exceed the 32767-character Windows environment-variable limit: PYTEST_CURRENT_TEST raises ValueError at setup and teardown even though the products under test behave correctly, and the giant ids flood -q reports. The skillscan linearity test additionally spawned one child process per chain. A fresh child spends 9-18 s importing the skills stack before the 4-168 ms scan, so its fixed 10 s timeout measured the import instead of the scan and flipped to subprocess.TimeoutExpired on a loaded host. Give the oversized cases short explicit ids, scan every chain in one child process fed through stdin, and enforce the linearity bound on each scan inside that child (5 s) while a 60 s outer timeout covers the import. |
||
|
IronMurphy
|
6103c7b6c2
|
fix(lark): publish the skill file and manifest atomically (#6683)
* fix(lark): publish the skill file and manifest atomically _write_skill_file and _write_manifest wrote their final paths with a truncating write, while the runtime files in the same module already go through _publish_bytes_atomically. A partial skill file or manifest is unparseable, and the manifest carries the content hash the install is verified against, so a half-written one cannot be repaired. Route both through the existing helper. * fix(lark): address skill publication review feedback --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
yetuge
|
97ed0e9797
|
fix(agents): skip content blocks with unhashable type values (#6664)
* fix(agents): skip content blocks with unhashable type values
has_visible_content and ParentContextSnapshot.from_state classified a
content block's type with set membership (`in {"text", "output_text"}`
plus the _MEDIA_BLOCK_TYPES frozenset). `type: []` / `type: {}` are
legal JSON from a provider, and hashing an unhashable value raises
TypeError instead of skipping the block:
- has_visible_content runs on every model response through the
llm-error-handling (_raise_for_empty_response), terminal-response and
model-length middlewares, so one malformed block turned a completed
response into a failed run instead of a skipped block;
- from_state builds the history every background subagent dispatch is
seeded from, so the same block failed the dispatch.
Probe the type by equality with tuples so a non-string type simply fails
the probe, mirroring #6614's fix for extract_response_text and the
isinstance guard clarification already uses for form field types. The
media constant becomes a tuple too, so the second probe in the same loop
cannot move the crash one line down.
Tests pin the RED-to-GREEN cycle at both call sites: unit-level
has_visible_content, middleware-level empty-response classification,
and the dispatch snapshot across human/ai/tool messages.
Fixes #6663
Signed-off-by: yetuge <2219677952@qq.com>
* fix(agents): warn when a snapshot skips a non-string-typed block
Review follow-up on #6664: with the malformed block now skipped
instead of raising, a provider that systematically emits non-string
types would degrade the child's history with nothing in the logs to
explain it. The elif chain cannot log on fall-through —
reasoning/signature and tool-use exclusions are deliberate and must
stay quiet — so the signal keys on the type not being a str, matching
the guard the review suggested.
has_visible_content keeps its silence on purpose: the empty-response
retry already surfaces the symptom there.
Fixes #6663
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
|
||
|
Jie Ji
|
bdc2d12367
|
fix(memory): prevent cross-conversation debounce starvation (#6672)
* fix(memory): prevent cross-conversation debounce starvation * fix(config): bump schema version for memory max wait |
||
|
ZJPex
|
65e8b6591b
|
fix(tools): bound deferred catalog regex execution (#6676)
* fix(tools): bound deferred catalog regex execution * fix(tools): clarify deferred regex limit diagnostics |
||
|
Jason Cat
|
d5e0a0a269
|
fix(tavily): contain malformed web search and fetch payloads (#6645)
* fix(tavily): contain malformed web search and fetch payloads web_search and web_fetch normalized the SDK's decoded 200 body without validating its shape: a `results`/`failed_results` container of the wrong type, a non-object entry, or a missing leaf field escaped as a raw TypeError/KeyError/AttributeError instead of the structured format error brave, serper, serply, sofya, tencent_wsa, unbrowse and groundroute return. Validate the container with one shared helper, skip non-object entries while preserving valid objects in order, normalize missing leaf fields to empty text, and coerce a non-string raw_content in web_fetch before truncating. Diagnostics carry only the malformed container's name and type, never the query, URL, key or payload values. backend/tests/test_tavily_response_shapes.py drives the real tools and SDK over an offline HTTPX MockTransport; it fails 32 of 36 cases on main and passes with this change, while test_tavily_tools.py stays green. * docs(readme): note Tavily payload containment Follow the Brave and GroundRoute README entries in the setup wizard step; the containment contract itself lives in community/tavily/AGENTS.md. * fix(tavily): normalize null search fields and test mixed fetch lists --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
IronMurphy
|
3c547e5ed7
|
fix(lark): publish the extracted runtime and launcher atomically (#6671)
_write_lark_cli_runtime_from_archive and _write_lark_cli_sandbox_launcher wrote their file with a truncating open and chmod 0o755 afterwards, so a crash mid-write left an executable fragment behind - for the extracted binary that file can be large. Publish through a temporary file plus os.replace, as the flow-state writer in the same module already does. |
||
|
yetuge
|
8a3350ac8a
|
fix(tools): publish the default agent's SOUL.md atomically (#6668)
* fix(tools): publish the default agent's SOUL.md atomically setup_agent persisted the default agent's global SOUL.md with a truncating write_text. The open truncates the previous file before the new content lands, so an interrupted setup — process kill, disk full, an exception while flushing — leaves a truncated persona file, and the prompt builder reads SOUL.md on every run, so the agent then silently runs with part of its instructions missing. The same file is already published atomically everywhere else: the custom-agent store stages it to a temporary and commits with os.replace, and #6618 replaced the identical truncating pattern for the Lark CLI runtime files the same day. The default agent's copy was the one remaining write of this file that could tear. Publish through a staged temporary plus os.replace, mirroring the store's _stage_temp commit, and clean the temporary up when the publish fails so no partial artifact survives. Tests: a failed publish (os.replace raising) keeps the previous SOUL.md byte-for-byte and leaves no temporary behind, a control pins that the old truncating write loses the content in the same failure, and the happy path writes the full file. Fixes #6667 Signed-off-by: yetuge <2219677952@qq.com> * fix(tools): keep SOUL.md's mode and clean the temporary on staging failure Review follow-up on #6668, both non-blocking suggestions that guard the publish itself: - NamedTemporaryFile creates the temporary with mode 0600, so the first publish made the global SOUL.md 0600 (the old write_text produced 0644 & ~umask) and every re-publish clobbered a mode an operator had set. Stat the target and chmod the temporary to that mode (0644 for a first publish) before the replace — the order #6618 codified for the Lark runtime files and atomic_write_extensions_config already follows. - A failure inside the staging block (handle.write or the implicit flush/close, e.g. ENOSPC) raised before the try started, orphaning the delete=False temporary in base_dir. One try/finally now spans staging and publishing, so neither phase can leave a temporary behind. The exact POSIX mode semantics were verified outside the Windows host (WSL Ubuntu: mkstemp is 0600, first publish is 0644, a re-publish keeps an operator-set 0640, and both failure phases leave no temporary); the exact-mode tests are therefore POSIX-gated, with a cross-platform test pinning that the chmod lands on the temporary before the replace and a staging-failure test that is red on the previous head. The shared-helper suggestion is left as a follow-up: folding six independent publishers into one helper is a cross-module refactor, not part of this bug fix. Fixes #6667 Signed-off-by: yetuge <2219677952@qq.com> --------- Signed-off-by: yetuge <2219677952@qq.com> |
||
|
Hyeonsang Cho
|
2ca0b79f73
|
fix(tui): keep a refused message in the composer (#6666)
* fix(tui): keep a refused message in the composer on_composer_input_submitted cleared the composer before _send_to_agent decided whether it could start a run. A prompt typed during an active run, while an interrupted run on the same thread was still stopping, or when the worker failed to start was replaced by a notice and survived only in input history. _send_to_agent now reports whether it started a run, and the submit handler puts the exact submitted text back with the cursor at its end, so Enter resends it. Accepted messages and local slash commands clear the composer as before. * fix(tui): keep a launch message that fails to start in the composer on_mount sent the launch message through _send_to_agent and ignored the result, so a worker start failure lost the message: it never reached the composer or input history. Route both the launch message and composer submissions through one _restore_draft helper. * docs(changelog): reference #6666 for the TUI refused-message fix * fix(tui): keep the palette closed over a restored slash draft Restoring a refused bare skill activation such as /tdd posted TextArea.Changed, which reopened the command palette, so Enter filled the skill instead of resending it. Remember the restored draft and keep the palette closed until it is edited. Restore the history copy of the text too, so a trimmed slash command matches what Up recalls. |
||
|
Storm
|
a68e3064e1
|
chore(deps): refresh lark-oapi lock to 1.7.3 (#6665)
Refresh the locked lark-oapi from 1.5.5 to 1.7.3 without touching the declared >=1.4.0 constraint, which already admits the new release. Deployments syncing from the lockfile move off the 1.6.x line, where a server-side protocol drift reportedly left long-connection clients connected but receiving no event pushes. Keeping the constraint as-is avoids shutting out downstream consumers pinned to older but working releases (1.5.5's WS event routing, including the callback processor map used by register_p2_card_action_trigger, is verified intact in source). - Refresh uv.lock only: lark-oapi 1.5.5 -> 1.7.3, specifier unchanged - Feishu channel, channel-connections, and thread-binding suites pass on 1.7.3 (111 passed, 1 skipped); ruff clean Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
xiaodu55
|
ea68e5b3d7
|
test(local-dev): normalize the upgrade script's MSYS path rendering on Windows (#6660)
* test(local-dev): normalize the upgrade script's MSYS path rendering on Windows nginx-local-conf.sh computes its repo root with pwd -P, which inside Git Bash yields the MSYS mount form (/tmp/... for the Windows temp dir) — correct for its bash-native consumers (serve.sh, nginx.sh) but never equal to the test's Windows-form Path.resolve(). Map the rendering back through cygpath -w before comparing, so the tests read and compare the rendered config on Git Bash hosts too. Fixes #6657 * test(local-dev): harden Git Bash path normalization --------- Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Yeager0204
|
782fed9135
|
feat(skills): show YAML load errors for owned custom skills (#6623)
* feat(skills): show YAML load errors for owned custom skills * docs: capture skill diagnostics with real gateway validation * fix(skills): clarify YAML diagnostic scope * docs: remove PR evidence directory --------- Co-authored-by: kwsYeager <331533331+kwsYegar@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
5116434712
|
fix(tui): keep every word of an unquoted --print/--json message (#6636)
* fix(tui): keep every word of an unquoted --print/--json message --print and --json took a single optional value and never read the positional words, so `deerflow --print summarize this repo` sent only "summarize", `deerflow summarize this repo --print` exited 2 asking for a message on a terminal, and with piped stdin the typed words were ignored. Both flags now take every following word and fall back to words given before them, as --cli does; a typed message wins over piped stdin. A message split around the flag is refused with exit 2, because argparse loses the word order between the two parts. * docs(changelog): reference #6636 * fix(tui): refuse --print and --json together Only one headless mode runs, so the other flag's words were dropped: '--print hello --json world' ran with 'hello', and '--print --json hello' asked for a message. Put both flags in a mutually exclusive group so argparse exits 2 for every combination. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Value Add
|
4beb6c4ea4
|
feat(community): add You.com web search provider (#6515)
* feat(community): add You.com web search provider Adds You.com as an opt-in community `web_search` engine, mirroring the existing community engines (tavily, brave, exa, firecrawl, groundroute). You.com runs its own web index and returns extracted page snippets per hit rather than a single meta description, so a research loop gets usable context without a follow-up fetch. The provider picks its endpoint by whether a key is configured, so it works out of the box and scales up when the operator adds one: - no key -> /v1/agents/search, keyless free tier - YDC_API_KEY -> /v1/search with X-API-Key, full Search API Endpoint and headers are always chosen together; the keyless endpoint rejects an auth header, so a stale key can never be sent there. Both return the same body shape, so one normalizer covers both. httpx only, no new dependency and no pyproject extra, so uv.lock is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint * fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY The doctor detail returned before any key check, so operators who had set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'. Reworded to match the sibling free-provider entries and stay accurate in both states. * fix(search): sweep youcom max_results onto the shared owner The single-owner gate that landed with #6563 flags this branch's private _coerce_max_results as a newly hand-rolled copy, so backend-unit-tests now fails after merging main. Follow the coordination note from review: delegate to deerflow.community.search_max_results.coerce_max_results, which owns the rejection profile (booleans, non-integral floats, OverflowError), and keep only the 1-100 mirror of You.com's server-side count clamp on top — clamping policy stays per-provider. The .inf regression test now goes through the public tool path and also pins the bool and non-integral-float shapes of the shared bar. * docs(tools): use the bare you.com platform link in config and docs Match every other provider entry: no other provider in config.example.yaml or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params from the three you.com/platform links. Integration attribution stays in the identifying User-Agent sent to You.com hosts. * test(youcom): cover shared max_results ownership and bounds Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters. --------- Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
RongJie G
|
b708145001
|
fix(runtime): require explicit resume identity capability (#6571)
* fix(runtime): require explicit resume identity capability Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com> * fix(runtime): guard key-only idempotency admission --------- Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
ZJPex
|
128dafd0ce
|
feat(tools): add opt-in literal keyword discovery (#6659) | ||
|
spud
|
75ce6f4920
|
fix(mcp): preserve server binding history across gateway workers (#6617)
* fix(mcp): preserve server binding history across gateway workers A peer Gateway worker could keep a stale pooled session after another worker deleted and identically re-added a stdio server, because the final effective MCP slice matched the revision it had already applied. Record a per-server lifecycle generation in the shared extensions config and consume it in the reader, discovery claim, and committed handoff so an observed generation change force-rebinds only the affected server. Legacy configs derive a stable in-memory token; deletions leave permanent tombstones. * fix(mcp): fail closed on an invalid mcpLifecycle instead of retrying An unverifiable mcpLifecycle made initialize_mcp_tools() unpublishable, so get_cached_mcp_tools() retried discovery forever and could keep respawning stdio servers. Abort discovery on an invalid ledger after conservatively retiring local MCP state; a fixed config recovers on the next call. Also documents the rolling-upgrade prerequisite: cross-worker ABA detection requires configuration writes to go through a lifecycle-aware writer. * refactor(mcp): share stdio fingerprinting across lifecycle paths Reuse lifecycle._stdio_fingerprints from the applied-revision builder so the connection-identity rule has a single implementation, and move the unusable-server diagnostic into the shared helper. Add a regression proving the Gateway Skills writer preserves mcpLifecycle byte-for-value, and document that DeerFlowClient.update_mcp_config can raise RuntimeError after the config was committed. * docs(mcp): condense lifecycle guidance under the AGENTS chain limit The effective mcp/AGENTS.md chain was 15 bytes under the 98,304-byte CI hard limit, so the standalone cross-worker lifecycle bullet pushed it over. Fold the mcpLifecycle and rolling-upgrade facts into the existing selective-reconciliation bullet and condense it, keeping the chain under the limit while preserving the contract. * docs: trim inherited agent guidance below CI size limit --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Dragon ze
|
e5d9a5f42d
|
refactor(community): give retrieval credential handling a single owner (#6638) | ||
|
Dragon ze
|
6709c58256
|
refactor(models): give reasoning capture plumbing a single owner (#6639) | ||
|
Hyeonsang Cho
|
b9b74d2338
|
fix(scripts): reject local BIND_HOST values nginx would misread (#6635)
* fix(scripts): reject local BIND_HOST values nginx would misread nginx-local-conf.sh checked only for address and hostname characters. BIND_HOST=0 rendered `listen 0:2026;`, which nginx resolves to 0.0.0.0 and binds on every IPv4 interface, and 1.2.3 bound 1.2.0.3. Values nginx rejects outright ([[fd00::1]], -foo) passed the serve.sh preflight and failed only after stop_all had torn down the running stack. Accept exactly a dotted-quad IPv4 address, an IPv6 address (brackets allowed around IPv6 only), or an RFC 1123 hostname. Returning to the loopback default removes a temp/nginx.local.conf rendered for an earlier value, and stop_all now says why it can quit a rendered-config nginx through the tracked config; a test pins the shared pid directive. * docs(changelog): reference #6635 * docs(changelog): match the zh reference spacing * fix(scripts): check every BIND_HOST hostname character before splitting labels read -r -a stops at the first newline, so is_hostname validated only the first line of a value like $'localhost\nbad' and the rest reached the rendered listen directive. Check the allowed characters across the whole value first, and cover multi-line, CR and tab values for each validator. |
||
|
creed
|
3461c12896
|
feat(jina): add process-local request admission (#6489)
* feat(jina): add process-local request admission Signed-off-by: 97three <2212371308@qq.com> * test(jina): cover closed admission owner loop Signed-off-by: 97three <2212371308@qq.com> * test(jina): retain admission across API redirects Signed-off-by: 97three <2212371308@qq.com> * fix(jina): drain redirect response cleanup before releasing admission Signed-off-by: 97three <2212371308@qq.com> --------- Signed-off-by: 97three <2212371308@qq.com> |
||
|
jiujiu
|
3234563db8
|
fix(utils): skip malformed response block types (#6614)
Co-authored-by: songjianglin <270247441+2017955725@users.noreply.github.com> |
||
|
IronMurphy
|
05297813ae
|
fix(lark): publish the CLI runtime files atomically (#6618)
* fix(lark): publish the CLI runtime files atomically install_shim wrote the shim, the launcher and the runtime marker with a truncating open. The sandbox chmods the launcher 0o755 and execs it, so a crash mid-write left an executable fragment, and the marker is parsed as JSON. Publish through a temporary file plus os.replace, as the storage backends already do. * style: drop the unused import ruff flags * fix(lark): publish runtime permissions atomically Set file permissions on temporary files before os.replace() to ensure the launcher, shim, and runtime marker are atomically published with correct permissions (0o755 for executables, 0o644 for the marker). This addresses reviewer feedback from PR #6591: - Permissions are now set before the atomic replace operation - Prevents brief window where files have incorrect permissions - The runtime marker now has mode 0o644 instead of 0600 (mkstemp default) Also updates test_lark_shim_atomic.py: - Fixes test_atomic_helper to match new function signature - Adds test_install_shim_publishes_correct_permissions to verify launcher/shim (0o755) and marker (0o644) permissions Co-authored-by: IronMurphy <hepushuo@gmail.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
NanPan
|
38c55c67f6
|
fix(streaming): report replay gap for future Redis Last-Event-ID (#6605)
* fix(stream): report replay gap for future Redis stream cursors * test(stream): future reconnect cursors report gap on live and ended runs |
||
|
Jason Cat
|
65a186187d
|
fix(workspace-changes): keep the package importable as a first import (#6602) (#6608)
`workspace_changes/api.py` imports `deerflow.runtime.user_context` for the AUTO sentinel (#6282). Importing that submodule runs the runtime package init, which imports `.runs` -> `worker` -> `deerflow.workspace_changes` back for `capture_workspace_snapshot` and friends, so importing `.api` as the package's first statement failed with ImportError while those names were still unbound. Resolve the re-export lazily via a module-level `__getattr__`, the pattern `deerflow/subagents/__init__.py` already uses. |
||
|
Nan Gao
|
35f6a98cd9
|
feat(dev): add a two-Gateway multi-instance harness script (#6613)
* feat(dev): add a two-Gateway multi-instance harness script There was no way to run the deployment.multi_instance topology locally: scripts/serve.sh pins one Gateway to port 8001, the local nginx config has a single upstream, and getting two processes past the multi-instance startup gate needs Postgres, Redis, a shared DEER_FLOW_HOME, shared auth secrets and a per-process DeerMem retrieval index by hand. scripts/dev_multi_instance.sh (make dev-multi / dev-multi-check / dev-multi-down) starts throwaway Postgres and Redis containers on loopback, renders a config from the developer's config.yaml (or config.example.yaml), and runs Gateway A on 8001 and Gateway B on 8011 with one shared home, one copy of extensions_config.json and generated AUTH_JWT_SECRET, DEER_FLOW_INTERNAL_AUTH_TOKEN and DEER_FLOW_CREDENTIALS_KEY values, plus an optional round-robin nginx on 2027. B starts after A answers /health: Alembic migrations are advisory-locked, but the LangGraph checkpointer/store setup() that follows is not. scripts/dev_multi_instance.py holds the testable parts: the config overlay (neutralizing and reporting settings the gate refuses), the secrets file, the nginx config, and `check`, which automates the scriptable cross-instance acceptance checks: readiness with stream_bridge ok, A-issued sessions and the internal token accepted by B, thread/upload/artifact visibility, a skill toggle seen by B, nginx balancing, and SSE join on B with Last-Event-ID resume on A. Tests pin that the rendered config passes the real _enforce_postgres_for_multi_worker gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): link the two-Gateway harness entries to #6613 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dev): keep the multi-instance harness inside its own state Address review findings on the two-Gateway harness (#6613): - Containers carry a deerflow.harness.state-dir label; up/down remove only containers of their own state dir and refuse those of another checkout or DEERFLOW_MI_STATE_DIR instead of force-removing them. - Containers are removed with `docker rm -f -v`, so the anonymous volumes the postgres and redis images declare no longer leak on every up/down. - `up` installs an EXIT trap once it starts containers, so any later failure (including B's port being taken after preflight) tears down A and both containers; start_gateway returns failures instead of exiting the script, and the Gateway is disowned so stopping it prints no job notice. - The overlay removes any checkpointer section (a postgres one would write checkpoints and the Store into the developer's database), points an explicit sandbox.ownership and a Redis checkpoint_cache at the harness Redis (keeping ownership timing), drops DeerMem storage_path (both spellings) and the blob_storage root so data lands in the harness home, gives a local AIO sandbox a state-dir-specific container_prefix, and reports a non-DeerMem memory backend it keeps sharing. - nginx paths are rendered as quoted string literals ('$' and control characters are refused), so a state dir with spaces works. - `check` skips the run-based SSE check when model discovery did not succeed, unless --with-llm, and enforces the SSE time budget per raw line so heartbeat-only traffic cannot hold it open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(scripts): keep the harness guidance under the AGENTS.md soft limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
ZJPex
|
7e0e3013d0
|
feat: return hit-centered history search excerpts with source offsets (#6008)
* feat: return history search excerpts near keyword matches * docs: use English for history search guidance and comments * docs: sync Chinese history search excerpt contract * docs: clarify history excerpt fallback for oversized matches * fix(history): skip oversized occurrences when selecting excerpts Continue scanning casefolded substring matches until an occurrence fits the 600-character excerpt, including overlaps. Cover later fitting hits through synchronous and asynchronous active/archive search and read tools, and update the documented contract. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Storm
|
588ad010c0
|
fix(mcp): stop persisting URL-sourced ResourceLink blocks and heal bricked threads (#6606)
* fix(agents): sanitize persisted URL content blocks before model calls
Threads whose checkpoints carry URL-sourced file blocks (written before
the MCP conversion-layer fix) fail every model call at serialization: the
Chat Completions translator rejects any file block carrying a url, and the
error handler answers each later turn with a non-retriable fallback. Add a
read-time sanitizer that rewrites those blocks in the model-bound request
view only, so poisoned threads heal on their next turn with no migration
and no user action.
- ModelContentCompatibilityMiddleware wraps the model call (sync and
async) and rewrites ToolMessage list content: URL-sourced file blocks
become "[Resource ({mime or unknown type}) available at {url}]" text
blocks, and image blocks whose scheme is not http/https/data get the
same placeholder. Fetchable image URLs, base64 payloads, text, and
non-ToolMessage messages pass through. The rewrite is order-preserving,
idempotent, and never writes state or mutates checkpoints, so artifact
capture and other state readers keep seeing the original blocks.
- Register on all three chains that replay checkpointed ToolMessages:
build_middlewares (lead; outer of the MODEL_PHYSICAL extension anchor so
its final-request guarantee holds), the SDK factory
_assemble_from_features (new step 13a; docstring updated and its stale
omission of TokenBudgetMiddleware fixed), and
build_subagent_runtime_middlewares (after DurableContext/summarization,
before the date/coalescer tail).
- Subagent coverage verified by tracing every subagent agent-creation
site: executor.py _create_agent is the only caller of
build_subagent_runtime_middlewares and serves both the task tool and the
durable batch worker (SubagentRuntime itself only carries dependencies),
and a wiring test now asserts the middleware is present in that
builder's output; the lead and factory chains gained the same wiring
assertions.
- Update the chain-order contract tests for the new slot: full-chain
order, loop-detection adjacency, custom loop-detection tail positions,
and the subagent chain count (22 -> 23).
* fix(mcp): stop emitting URL-sourced file blocks from ResourceLink results
MCP ResourceLink conversion produced a URL-sourced file content block for
every non-image link, which LangChain's Chat Completions translator rejects
at serialization time. Once such a block was checkpointed in the
ToolMessage, every subsequent turn of the thread failed at model-call time,
permanently bricking the conversation.
- Downgrade every ResourceLink that is not an http(s) image to a short text
placeholder carrying the resource name, MIME type, and the
already-virtualized URL, so host paths never leak and the model still
learns a resource was produced and where it lives.
- Preserve each downgraded link as structured data in the tool-message
artifact under resource_links (merged with structured_content when
present) for artifact capture and future MCP Apps rendering.
- Keep http(s) image links as image blocks; embedded base64 payloads, text
path rewriting, and the error-to-ToolException path are unchanged.
- Re-contract the file-migration tests to the new behavior and document
the conversion contract in the MCP AGENTS.md.
* docs(middlewares): add ModelContentCompatibilityMiddleware to the chain inventory
Insert the read-time sanitizer at its actual lead-chain position (after the
token-budget guard, before the custom-middleware injection point) and renumber
the tail entries 34-39 to 35-40.
* fix(agents): rewrite file blocks on url key presence, not truthiness
langchain-core's OpenAI translator raises for ANY file block carrying a
url key — an empty string or None included — so the read-time sanitizer
must not let such blocks through just because the value is falsy. A
persisted block with url="" or url=None would still brick every later
model call in the thread.
- Gate the file-block rewrite on "url" in block instead of a truthiness
check; a non-empty string URL keeps the
"[Resource (mime) available at url]" placeholder, while an empty or
None URL gets a location-less "[Resource (mime)]" placeholder, so no
unusable location is advertised to the model.
- Pin both shapes with new middleware tests (url="" and url=None file
blocks are both rewritten to text). The image branch is unchanged: an
empty-image case fails provider-side with a 400, not a bricked thread.
* fix(mcp): keep data URIs out of model text and resource_links
A data: scheme ResourceLink fell into the generic placeholder branch,
inlining megabytes of base64 into the model-visible text and into
resource_links, which is checkpointed state — against the repo policy
that data:/blob: URIs never enter state.
- Special-case data: links in _convert_call_tool_result: the
model-visible placeholder becomes "[Resource: name (mime) embedded
inline]" and the raw URI is not added to resource_links at all, so the
payload can neither bloat the context window nor persist in the
checkpoint.
- Pin the behavior with a conversion test asserting both the placeholder
wording and the absence of the payload/URI from every emitted block and
the artifact.
- Add the missing passthrough pin for an HTTPS:// uppercase-scheme image
link: scheme matching is case-insensitive, so the link stays an image
block (the block's URL validation normalizes the scheme to lowercase).
* fix(tools): consume resource_links in extract_artifacts_from_result
The MCP conversion layer preserves downgraded ResourceLinks as
{"name", "uri", "mime_type"} entries under the tool message
artifact's resource_links key, but the artifact extractor only read
structured_content — a result whose only artifact signal was
resource_links produced no entries and the file silently vanished from
the artifacts panel.
- Add a fourth extraction path: each resource_links entry becomes a file
artifact entry, reusing _is_referenceable_url like the
structured-content and content-block paths, so data:/blob: and other
unresolvable URIs stay rejected per existing policy. The entry is named
after the link when a name is present, falling back to the URI's final
segment; a string mime_type is carried through.
- Integration-test a result whose only signal is resource_links, plus the
referenceability gate (data:, ui://, and absolute-path URIs in one
artifact).
* docs(mcp): update the ResourceLink conversion contract bullet
Mention the new artifact-registry consumer and the data: special case so
the bullet stays accurate: uppercase http(s) schemes match
case-insensitively, data: URIs get the embedded-inline placeholder and
are dropped from resource_links, and extract_artifacts_from_result turns
resource_links into registry entries through the shared
referenceability gate.
* refactor(tools): add the canonical resource placeholder formatter
Both the MCP conversion layer and the read-time
ModelContentCompatibilityMiddleware downgrade model-incompatible resource
references to a text placeholder, but each built its own literal. Extract
the single formatter they will share so a thread healed at read time shows
the model the same placeholder shape a fresh conversion produces; the name
segment is optional because persisted content blocks carry no resource name.
* fix(mcp): keep host paths out of ResourceLink placeholder text
Review follow-up on the ResourceLink conversion contract. The placeholder
embedded the resolved URL unconditionally, so a link outside the user-data
tree (or converted without thread context) leaked a raw host path — a
file:// URI or bare filesystem path — into model-visible text, violating
US-16. Compute the resolved URL's scheme and withhold the location segment
when it spells out a host filesystem location (empty scheme, file, or a
single-letter Windows drive scheme); virtual paths and remote schemes keep
their location. The unresolved URI still lands in the artifact channel,
which the model never sees.
- add _model_visible_link_location gate and route the generic placeholder
through it, re-contracting the outside-user-data and no-context tests to
the location-less shape and pinning the Windows-drive case
- treat blob: like data: as non-referenceable inline input: drop its raw
URI from checkpointed resource_links and emit the location-less
placeholder instead of the mismatched 'embedded inline' wording, and fix
the comment that claimed blob: was already excluded
- replace every inline placeholder f-string in the ResourceLink branch
with the shared resource_placeholder_text formatter so the conversion
layer and the read-time middleware emit byte-identical shapes; data:
links now render in the canonical location-less form
- sync the ResourceLink conversion contract bullet in mcp/AGENTS.md and
the _convert_call_tool_result docstring with the host-path suppression
rule and blob: handling
* fix(agents): sanitize URL content blocks on any message role
Address the second-pass review of the request sanitizer with two
changes that share the same code path.
- Adopt the canonical resource_placeholder_text formatter from
deerflow.tools.resource_placeholder instead of the module-local
placeholder helpers, so the read-time rewrite and the MCP conversion
layer emit byte-identical placeholder text from one definition.
- Widen the rewrite from ToolMessage-only to every request message with
list-form content. User uploads are verified to produce base64 blocks,
but imported or cross-client history can carry a URL-sourced file
block on a HumanMessage or AIMessage, and the Chat Completions
serializer rejects such a block regardless of the role carrying it.
The rewrite stays request-view-only: state and checkpoints are never
mutated, and base64/file-id blocks and string content pass through
untouched on any role.
* docs(agents): state the ModelContentCompatibility placement contract
The comment above the registration claimed the sanitizer sits just
before Clarification. In the lead chain it is actually followed by the
custom/extension injection point and the whole terminal tail
(terminal-response, finish-reason, and the always-last
ClarificationMiddleware). Describe the real contract: after every
built-in content transform, before extension injection and the
terminal tail.
* docs(middlewares): clarify provenance stamps for request-view-only rewrites
The message-provenance rule read as if every rewrite needs a
provenance_kwargs stamp. Add one line noting the exception: rewrites
that only modify the model-bound request view and never enter state
need no stamps, citing InputSanitizationMiddleware,
DanglingToolCallMiddleware, and ModelContentCompatibilityMiddleware as
precedents.
* fix(agents): keep host paths and data URIs out of healed placeholders
The read-time ModelContentCompatibilityMiddleware rewrote URL-sourced
file blocks and non-fetchable image blocks into placeholders that
inlined the block's URL verbatim. Pre-fix checkpoints — exactly the
threads this middleware exists to heal — can carry a file:// URI, a bare
or Windows host path (links that never resolved to virtual paths), or a
data: URI embedding megabytes of base64, so healing leaked deployment
filesystem layout into prompts (US-16) and could inject the whole inline
payload into model-visible text.
The conversion layer already suppresses those forms via
_model_visible_link_location. Move that gate into the shared
deerflow.tools.resource_placeholder module as model_visible_location
(now also collapsing data:/blob: URIs) and route both middleware
branches through it, so both layers apply identical suppression rules
with one definition. Conversion behavior is unchanged: its data:/blob:
branch short-circuits before the gate, and the gate covers those schemes
for defense in depth.
- resource_placeholder.py gains model_visible_location alongside the
shared formatter; mcp/tools.py drops its private copy and imports it.
- The middleware's file and image branches pass the URL through the gate
before building the placeholder; module docstring documents the rules.
- Tests: gate unit tests (virtual/remote visible; file://, bare/Windows
host paths, data:/blob: withheld) and middleware regression tests
pinning that a pre-fix data: payload, file:// URI, and Windows drive
path heal to location-less placeholders.
- AGENTS.md bullets in mcp/ and middlewares/ state the shared gate.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(agents): downgrade malformed image URLs instead of raising in the sanitizer
The read-time ModelContentCompatibilityMiddleware gated persisted image
blocks on urlparse(url).scheme, but urlparse raises ValueError on
structurally invalid URLs — a bad IPv6 literal, invalid netloc
characters — instead of returning an unusable scheme. Pre-fix
checkpoints and imported history can carry exactly those URLs, and the
block stays in the checkpoint, so every later model call in the thread
raised again: the sanitizer re-bricked the very threads it exists to
heal, only with a framework-side exception instead of the langchain
serialization error.
Catch ValueError and treat the URL as non-fetchable, so the block
degrades into the same text placeholder as any other unresolvable
scheme (the shared model_visible_location gate already swallows
ValueError, so the location segment is withheld as well).
- wrap the image branch's urlparse call in try/except ValueError and
fall back to an empty scheme, keeping http/https/data pass-through
behavior unchanged for well-formed URLs
- add a regression test pinning that an image block with a bad IPv6
literal heals to the location-less placeholder without raising
Co-Authored-By: Codex <noreply@openai.com>
* fix(tools): dedupe artifact refs across extraction sources
One downgraded ResourceLink produced two registry entries: the typed
resource_links channel added an entry for the URI, and the free-text
scan over the conversion layer's placeholder text — which embeds the
same virtual path or remote URL — matched it again. The two handles
pointed at one resource, spent double of the render budget, and showed
the model two handles for a single artifact.
extract_artifacts_from_result now tracks seen real_refs across all
sources and keeps only the first entry per ref. Extraction order makes
the typed sources (structured content, resource_links, content blocks)
win over the free-text scan, so the surviving entry keeps the link's
display name and mime type.
- route all six entry-producing sites through a local _add_unique
helper keyed on real_ref; per-source behavior is unchanged
- document the cross-source dedup in the function docstring
- add a regression test covering a virtual-path and a remote-URL
placeholder together with their resource_links entries
Co-Authored-By: Codex <noreply@openai.com>
* docs(mcp): state the intentional data: image downgrade asymmetry
The conversion layer downgrades image/* data: ResourceLinks to
location-less text placeholders, while the read-time middleware lets
persisted data: image blocks pass through to the model. The two
contracts looked contradictory but are deliberately layered: the
conversion layer keeps new inline payloads out of checkpointed state,
and the middleware only preserves blocks already occupying one.
- mcp/AGENTS.md's ResourceLink conversion contract and the conversion
comment in mcp/tools.py now state the image/* data: downgrade and
why it differs from the read-time pass-through
- middlewares/AGENTS.md's ModelContentCompatibilityMiddleware entry
states the symmetric read-time rule
- pin the behavior with an image-mime data: ResourceLink conversion
test (location-less placeholder, nothing in the artifact channel)
Co-Authored-By: Codex <noreply@openai.com>
* docs(backend): compact AGENTS.md guidance to fit the chain byte budget
The branch's new contract bullets pushed the middlewares and mcp
effective AGENTS.md chains past the 98304-byte AG002 hard limit (the
sandbox/subagents chains on main were already over it). Compact the
guidance without dropping contracts:
- middlewares item 34 points at the MCP ResourceLink conversion
contract and the module docstring instead of restating both
- the mcp ResourceLink conversion contract bullet is tightened
- backend/AGENTS.md merges the File Upload and Context Summarization
bullet sprawl, folds the stray ATX test note into the upload outline
rule, drops the Nginx routing block the root guide already covers,
merges the duplicate MCP-task pointer bullet, compacts Vision
Support, and scopes the Project Structure tree to the backend
subtree (the root guide maps the full repo)
Co-Authored-By: Codex <noreply@openai.com>
* fix(agents): gate the file-URL downgrade on provider support and neutralize placeholder metadata
Two review findings on the read-time sanitizer:
- The file-block downgrade ran for every provider, but only Chat
Completions rejects URL-sourced file blocks: the OpenAI Responses API
(input_file file_url) and Anthropic (document url source) accept
http(s) document URLs, so a persisted block created for those
providers lost its document from the model request. Keep the block
when the selected model serializes http(s) file URLs natively; every
other URL shape (non-http schemes, empty/None, malformed) still
degrades to the placeholder.
- The placeholder interpolated persisted MIME/location metadata
verbatim, but the middleware runs after the input/tool-result
sanitization passes and historical URL blocks never went through them
(non-text content is left untouched), so values carrying
<system-reminder>-style tags or user-input boundary markers reached
the model raw. Route the interpolated metadata through the shared
neutralize_untrusted_tags guard in both the file and image branches.
Tests: provider gating (Responses/Anthropic preserve, Chat Completions
still downgrades, non-http and malformed URLs downgrade even for
capable providers) and metadata neutralization (injection tags
HTML-escaped, boundary tokens neutralized, file and image branches).
Co-Authored-By: Codex <noreply@openai.com>
* fix(tools): require a directory boundary before exposing virtual locations
model_visible_location admitted any URL starting with the virtual
prefix, so a sibling like /mnt/user-data-backups/ops/private/report.pdf
was treated as a virtual path and its host location reached
model-visible placeholder text — defeating the US-16 host-path
suppression the gate exists for. Require the exact virtual root or a
slash-delimited descendant; siblings now collapse to the location-less
placeholder like any other host path. Both the file and image
downgrade branches share the gate, and the conversion layer applies
the same rules to fresh results.
Regressions: gate unit tests (exact root and descendants stay visible;
/mnt/user-data-backups and /mnt/user-database siblings withheld) and
middleware tests pinning the location-less placeholder for both
branches.
Co-Authored-By: Codex <noreply@openai.com>
* docs(agents): scope the placeholder artifact-visibility claim to the conversion layer
The module docstring claimed the healed placeholder keeps the location
visible to the artifact free-text scan and that the link survives in
the structured artifact channel — true for the conversion layer, whose
placeholders are checkpointed, but not on the healing path: the request
view never reaches artifact capture, and no resource_links entry is
created for a downgraded block. Spell out what actually happens here:
a referenceable original block (http(s) URL or absolute host path) is
still captured, from the capture-time scan of the state block rather
than the placeholder, while a non-referenceable one (file://, data:,
blob:) yields no entry and survives only in the untouched checkpoint.
Co-Authored-By: Codex <noreply@openai.com>
* docs(mcp): clarify healing and artifact preservation contracts
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Jason Cat
|
d37616e659
|
test: route four unguarded symlink fixtures through symlink_or_skip (#6619)
* test: route four unguarded symlink fixtures through symlink_or_skip `tests/support/symlinks.py` was added in #5414 so suites that need a real symlink skip cleanly when the host cannot create one. On a stock Windows host (no Developer Mode, not elevated) `Path.symlink_to` / `os.symlink` raise `OSError: [WinError 1314]`, and four fixtures still called them directly, so they reported hard errors instead of skipping: - tests/test_shared_reset_marker.py::test_marker_path_follows_a_symlinked_config - tests/test_view_image_middleware.py::test_host_image_copy_is_scoped_to_current_user_and_thread - tests/test_review_skill_package_tool.py::test_review_skill_package_rejects_symlink_planted_in_callers_outputs - tests/test_channel_file_attachments.py::TestInboundFileSandboxPerms::test_make_inbound_file_sandbox_readable_swap_after_lstat_does_not_follow_symlink These were the only symlink-related failures in a Windows run of the full backend suite; the other bare symlink call sites there are already guarded by a `skipif` or a `try/except OSError`. The view-image test plants its symlink after its scoping assertions, so those still run before the skip. The inbound-attachment test stages a symlink from inside a monkeypatched `os.lstat` to exercise a TOCTOU race; a host that cannot create a symlink cannot stage that race at all, so skipping is the only honest outcome. * test: make channel symlink race fixture portable --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
db7340d2fa
|
fix(security): add DEER_FLOW_CREDENTIALS_KEY and wire the channel credentials cipher (#6611)
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
xiaodu55
|
e3f21ce105
|
test: fix two Windows-host failures (subprocess encoding, budget-boundary epsilon) (#6627)
* test: fix two Windows-host failures (subprocess encoding, budget-boundary epsilon) - test_env_file: pin PYTHONIOENCODING=utf-8 in the stripped child env; without it the child writes stderr in the host's ANSI codepage while the UTF-8-mode parent decodes it, the reader thread dies on the first non-ASCII byte, and stderr comes back None; - test_jina_retries: give the budget-cap assertion a 1e-9 tolerance — deadline = now + budget rounds, so the cap lands a few ulps above the exact 0.05 (observed 0.05000000000291038). Fixes #6626 * test: pin both sides of the capture and use an uptime-proof budget tolerance - test_env_file: the parent's capture decodes with the host locale when pytest runs without the Makefile's UTF-8 pins, so the child-side pin alone mirrored the crash rather than eliminating it. Pin the capture explicitly (encoding=utf-8, errors=replace, per the test_local_sandbox_encoding.py precedent) and carry both suite pins in the child env (PYTHONUTF8=1 + PYTHONIOENCODING=utf-8). - test_jina_retries: the float error scales with loop-time magnitude (host uptime), not with 0.05; a 1e-9 absolute slack leaves a ~389-day- uptime flake window. Use a relative tolerance instead. --------- Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com> |
||
|
haochen
|
8377aa3d39
|
fix(support-bundle): retain diagnostics when dotenv cannot be read (#6536)
Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
ordinary being
|
737492d4e1
|
fix(runtime): drain staged-terminal runs on shutdown before checkpointer teardown (#6620)
* fix(runtime): drain staged-terminal runs on shutdown before checkpointer teardown RunManager.shutdown() selected only pending/running records, but with an event store a run stages its terminal status in memory and keeps finalizing: journal flush, delivery receipt, workspace scan, the duration checkpoint write and the deferred terminal commit, all while terminal_commit_pending is set. A Gateway shutdown landing in that window did not await the run, so langgraph_runtime could close the checkpointer before the duration write landed - the #3373 class the drain exists to prevent. Drain those records too, using the existing _awaits_terminal_commit state that _renew_leases already relies on; await them without cancelling (a cancel would skip their terminal tail) and never overwrite their staged terminal status with interrupted. * fix(runtime): surface shutdown terminal finalizer failures Cover timed-out staged terminal runs for both commit barriers and terminal outcomes. Log failed finalizers with their run ID, staged status, and traceback while preserving status and completing the drain. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Jason Cat
|
f623e64e07
|
fix(acp): resolve a bare agent command through PATH before spawning (#6598)
* fix(acp): resolve a bare agent command through PATH before spawning `invoke_acp_agent` handed `acp_agents.<name>.command` straight to the ACP SDK, which spawns it with `asyncio.create_subprocess_exec`. That API applies no `PATHEXT` lookup on Windows, so a bare npm launcher (`npx`, `mcode`) raised `FileNotFoundError` although its `.cmd` shim was on `PATH`. The documented `command: npx` and `command: mcode` shapes could not start, and the tool reported "was not found on PATH" for an installed launcher. Resolve the configured command with `shutil.which` before spawning — the way the MCP Python SDK normalizes stdio server commands — and keep the configured value when nothing matches, so the existing install guidance still fires. Fixes #6597 * fix(acp): resolve the ACP agent command off the event loop `_resolve_agent_command` calls `shutil.which`, which stats the filesystem. Calling it inline in the async `_invoke` tripped the strict BlockBuster gate (`BlockingError: Blocking call to os.access`) for every command that resolves, and would block the Gateway event loop in production. Offload the lookup with `asyncio.to_thread`, matching the neighbouring `_get_work_dir`, `_build_acp_mcp_servers`, and `_format_invocation_error` calls. The blocking-io anchor configured `command="test-agent"`, a name that resolves nowhere, so `shutil.which` short-circuited before `os.access` and the gate never covered the new call. Point it at `sys.executable` so the lookup is really exercised, and cover the unresolvable-guidance branch end-to-end with a new test. Refs #6597 * fix(acp): keep the configured command in ACP remediation text Review follow-ups on the PATH resolution: - Only bare names are resolved. A configured path is interpreted by the spawn against the per-thread ACP workspace, so resolving it against the Gateway's working directory could pick a different file or hand the spawn a path that then fails. - `_format_invocation_error` receives `agent_config.command` again. A command can resolve on PATH and still fail to launch (a broken shebang, for example), and reporting the resolved path made "was not found on PATH" false and dropped the `codex-acp` hint. - The blocking-io anchor uses a bare name backed by a shim on a controlled PATH instead of `sys.executable`, which the bare-name scoping no longer resolves. Tests: the pass-through assertion in `test_invoke_acp_agent_uses_fixed_acp_workspace` pins PATH, and new coverage covers the bare-name scoping and the spawn-failure message. Refs #6597 * docs(acp): correct the path-resolution rationale A configured path is resolved by the spawn in its own process, so the pass-through reason must not claim it is resolved against the ACP workspace: that only holds on POSIX, while Windows resolves a relative executable against the calling process's directory. State the platform-neutral reason instead. Refs #6597 * fix(acp): resolve the launcher against the agent's own PATH The ACP SDK merges acp_agents.<name>.env over its inherited environment at spawn time, so a configured PATH, not the Gateway's, decides which launcher the agent would find. Resolve a bare command against that effective PATH, and use the Gateway's only when the agent does not override it. Return a successful lookup as an absolute path: a relative PATH entry makes shutil.which yield a result relative to the Gateway's cwd, while the spawn runs with cwd set to the ACP workspace, where that path resolves elsewhere or not at all. Regressions: a real-subprocess test with same-named launchers in the Gateway and agent PATHs (asserting which one ran), and a real-subprocess test whose PATH holds a relative directory while the ACP workspace is elsewhere. Both are red on the pre-fix code. |
||
|
yang rui
|
a333ac3189
|
feat(search): add Webz news search provider and setup integration (#5880)
* feat(search): add Webz news search provider and setup integration * fix(webz): honor caller result limits and clarify page counts * fix(frontend): render Webz search source links * fix(frontend): skip malformed search source entries |
||
|
yangfucheng123
|
3b8141aa43
|
fix(community): Fail closed on delegated fetch backends to close SSRF gap (#6531)
* Fail closed on delegated fetch backends to close SSRF gap * fix(community): resolve delegated-fetch credentials from one config snapshot Address review feedback on the fail-closed SSRF guard: - firecrawl/fastcrw client construction no longer re-reads get_app_config(), so the API key and endpoint come from the same revision the endpoint screen validated (prevents cross-revision credential misrouting). - fix the browserless test double to accept the new two-argument _get_browserless_client signature. - document the check-only (DNS rebinding) caveat on validate_delegated_backend_url. - add a CHANGELOG upgrade note for the default self-hosted backend break. * fix(doctor): consolidate delegated fetch backend checks and deployment guidance --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
四野
|
b398921538
|
fix(skills): preserve Unicode separators in JSONL history (#6615) | ||
|
Jason Cat
|
8834a759b8
|
fix(groundroute): contain malformed web search and fetch payloads (#6603)
* fix(groundroute): contain malformed web search and fetch payloads web_search and web_fetch assumed the GroundRoute 200 body was an object with a `results` list of objects, and read it outside the request try/except. A body that arrived as a bare list, as an object with a non-list `results`, or with non-object entries raised AttributeError / KeyError out of the tool boundary, so the tool call failed with a stack trace instead of the provider's format error. Validate the payload before normalizing it, following the containment rule already applied to brave (#6550), serper, serply, sofya, tencent_wsa and unbrowse: a non-object payload, a non-list `results` container, or a non-empty list holding no objects returns the structured format error, non-object entries in an otherwise valid list are skipped in order, and missing, null or empty results keep returning "No results found". * fix(groundroute): coerce non-string fetch fields web_fetch sliced the fetched page's `content` (falling back to `snippet`) to the snippet limit without checking its type, so a 200 body whose result object held a number, boolean, or object there raised TypeError / KeyError out of the tool boundary -- the same escape the previous commit closed for the container shape. A null title also rendered as "# None". Coerce non-string content to text before truncating, and render a missing or non-string title as empty, matching the coercion sofya and unbrowse already apply to their fetch fields. * docs(groundroute): scope the falsy-title note to actual behavior The guidance said a non-string title renders as empty, but the code only empties falsy titles; a truthy non-string title still renders as its text form via the f-string. Say "falsy" instead. |
||
|
hataa
|
47d8ff13b2
|
feat(authz): account disable/enable across every authentication surface (#4063 gap 3) (#6566)
* feat(authz): account disable/enable across every authentication surface (#4063 gap 3) Vertical slice 1 of the user-lifecycle work: an operator can suspend an account and every authentication surface rejects it, while the row — credentials, role, OAuth linkage — is retained so re-enabling restores exactly what was suspended. - users.disabled (NOT NULL, server default 0) via migration 0036_user_disabled; User/UserResponse carry and expose it. - Enforcement at all four surfaces: password login (no credential comparison happens), JWT session resolution (401 ACCOUNT_DISABLED — sessions die at the next request), PAT resolution (the generic 401, no oracle), OAuth provisioning (403 with a contact-admin message; the link is kept so re-enable restores access). - SQLiteUserRepository.set_disabled: single-column, serialized write (same idiom as update_system_role) with the last-ACTIVE-admin guard — disabled admins cannot authenticate, so they cannot restore management access; re-enabling is always allowed. Field-scoping completed both ways: the credential writer mirrors and never writes disabled, so a stale password-change snapshot cannot re-enable a suspended account. - PATCH /api/v1/admin/users/{id} becomes a partial account update (system_role and/or disabled), each field routed to its field-scoped writer; empty bodies 422. Tests 11->18: route (disable/re-enable, last-active-admin 409, empty 422, combined update), real-repo (field isolation both directions, disabled-admin-not-active, guard inside the serialized write, re-enable always), login rejection with the CORRECT password (only the disabled gate can reject) plus resolver-visible flag. Mutations verified red: guard removed (M19), login gate removed (M20 — after hardening the test with a real hash so the gate is load-bearing). * fix(authz): portable boolean default, alternate-authenticator suspension, active-admin role guard Review round 1 on #6566 — three P1s: - [PG default] sa.text("0") is a raw integer expression PostgreSQL rejects as a BOOLEAN default, breaking startup migration and fresh create_all bootstraps alike. Both the migration and the ORM model now use sa.false()/false() — dialect-rendered, portable. - [alternate authenticators] WebSockets bypass AuthMiddleware, so a disabled user's still-valid cookie could open a NEW browser/stream connection and drive the retained session; the standalone LangGraph authenticate callback accepted the same suspended JWT. Both now route through a shared post-lookup validator (validate_resolved_session_user in deps.py — token_version + disabled in one place) that the HTTP dependency also uses, so a lifecycle change lands everywhere at once. - [active-admin role guard] update_system_role counted every admin row, so 'disable B, then demote self' — or a combined {system_role, disabled} self-update — could strand the deployment with zero administrators able to authenticate or re-enable anyone. The demotion guard now counts ACTIVE admins only (disabled admins cannot authenticate), under the same serialization boundary. Tests 18->22: shared-validator verdicts, WS admission after suspension (real token, real repo), LangGraph-callback admission after suspension, disable-then-demote-self + combined self-update both rejected (fake-repo route level and real-repo update_system_role level). Mutations verified red: helper's disabled check removed (M21 — three admission tests fail), active-admin predicate reverted to all-admins (M22 — repo-level strand scenario fails). 366 tests green on the affected surface. * fix(migrations): chain 0036_user_disabled after upstream 0037 The maintainer main-merge brought 0037_project_document_summaries, which branched from 0035 independently of this PR's 0036 — alembic now sees two heads and every engine bootstrap fails. Chain 0036 after 0037 (single head; revision numbers are labels, the graph is what matters). * fix(migrations): renumber user_disabled to 0038 and follow the chain-test convention Upstream landed its own 0036 (run_idempotency_request) with 0037 chained after it while this PR was in review — my 0036_user_disabled collided on the number and my interim chain-fix (0036 revises 0037) left the graph in a state upstream's chain tests reject (heads == [0037] hardcoded). - 0036_user_disabled -> 0038_user_disabled, revises 0037_project_document_summaries: single head 0038. - Chain tests follow the repo convention (each migration PR appends the new head): 0032/0035 gain the DOC_SUMMARIES link and CURRENT_HEAD = 0038; 0037's single-head assertion moves to CURRENT_HEAD with its own-revision link check unchanged. * fix(migrations): pin the remaining chain tests to the 0038_user_disabled head The 0027/0028, 0029 and 0034 chain tests still pinned the single head to 0037_project_document_summaries (and 0034 asserted the head's parent was 0036_run_idempotency_request), so the renumbered lifecycle migration broke two unit-test shards. Follow the chain-test convention: every pinned head moves to 0038_user_disabled and 0034 walks the chain through 0037. * fix(authz): enforce account suspension on the internal-auth dispatch surface Round-3 review finding: the internal-auth principal is synthesized from trusted headers without a users-row lookup, so a suspended user with an active channel binding kept driving the agent, and scheduler-owned runs (their fabricated in-process requests skip AuthMiddleware entirely) kept launching. Two gates now enforce the lifecycle there: - AuthMiddleware rejects owner-bound internal calls whose users row is disabled (401 account_disabled) at the single gate every internal HTTP call passes through. Auth-disabled mode is exempt (no suspendable accounts, and its owner header carries the synthetic identity), and owner ids without a users row stay allowed — a nonexistent account cannot be suspended. - start_run re-asserts it at run admission after the owner is resolved for attribution: the scheduler records the rejection as the occurrence's launch error and no run record is created. Also addresses the remaining round-3 nits: drop the inline token_version check duplicated by validate_resolved_session_user, correct the local_provider comment (login never surfaces ACCOUNT_DISABLED), refresh the stale 0038 docstring header, and restore the deps module caches in the two suspension tests instead of leaking a closed engine. * fix(auth): carry the disabled field on the frontend synthetic users UserResponse now always includes disabled, so the auth-disabled contract e2e (gateway /auth/me deep-equals the frontend synthetic user) failed without the field on AUTH_DISABLED_USER. Add disabled to both synthetic users and declare it optional on the zod schema — absent still parses as not disabled for pre-gap-3 backends. * fix(authz): log when the internal-owner suspension check is skipped The middleware gate's users-lookup failure path silently degraded to allow; keep the fail-open behavior but leave a WARNING with the exception, so a store that cannot be read is observable in production logs instead of invisible. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
NanPan
|
29d654e86d
|
fix(checkpoints): treat corrupt Redis history cache entries as misses (#6592)
* fix(checkpoint): treat malformed Redis history cache entries as misses * test(checkpoint): cover malformed Redis history cache entries * style(checkpoint): format corrupt cache regression test * test(checkpoint): cover non-dict cache entries and pinned Ruff formatting * fix(checkpoint): include cache key in malformed-entry diagnostics * docs: trim inherited agent guidance below size limit --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
f8b419ab38
|
fix(scripts): bind local make dev and make start to loopback (#6587)
* fix(scripts): bind local make dev and make start to loopback serve.sh and backend/Makefile started the Gateway with --host 0.0.0.0, nginx.local.conf listened on every interface, and Next.js kept its all-interfaces default outside Windows, so a local stack on a LAN or VPN exposed ports 2026, 8001 and 3000, including /setup before the first admin existed. The Docker stack and the README already document a loopback-only default. Bind the Gateway and frontend to 127.0.0.1 and keep nginx on loopback unless BIND_HOST is set, the variable the Docker stack already honors. nginx-local-conf.sh renders a copy listening on BIND_HOST, and serve.sh resolves it before stopping anything so an invalid value cannot tear down a running stack. * docs(changelog): reference #6587 * docs(changelog): separate the loopback entry from the next bullet --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
yetuge
|
ab9920f25f
|
fix(sandbox): frame remote search records on LF only (#6595)
* fix(sandbox): frame remote search records on LF only
BoxLite, OpenSandbox and Tenki sliced find/grep output with str.splitlines(),
which also ends a line at a bare carriage return, form feed, vertical tab,
file/group/record separator, next-line, and U+2028/U+2029. Every one of those
is legal inside a Linux filename and inside matched text, and the shared
parser already documents the opposite contract ("Split on \"\n\" only:
splitlines() would also split on characters that are legal in Linux
filenames").
Effects on these three providers, both silent:
- A path such as notes\x0bdraft.txt is reported as two unrelated paths, so
glob() returns a path that does not exist (dropping the real one) and
grep() reports the match against that wrong path.
- A matched line containing U+2028 -- which occurs inside real JavaScript and
JSON string literals -- is truncated at that character.
Split on "\n" only, matching LocalSandbox, the AIO backend and E2B (#6573).
The colon half of #6573 is deliberately not included: it needs grep --null,
and these providers restrict their command to busybox-portable flags by
design. BusyBox grep rejects --null/-Z (it prints its usage line instead of
searching), so adding the flag here would break every grep on a BusyBox
image. Splitting the record from the right is not a substitute because
"path:12:needle: hello:world" is genuinely ambiguous without a NUL separator.
Tests: a cross-provider file pins the framing through each provider's own
transport seam (canned output, so it runs on any host) plus real-shell cases
for a separator-named file and separator-bearing matched text. 81 of them fail
on main and pass here; the provider suites are unchanged at 268 passed.
Verified with PYTHONUTF8=0.
Fixes #6594
* test(sandbox): make the real-shell fixtures POSIX-correct
Both real-shell tests were written against a Windows host, where _RS_POSIX
skips them, so neither had ever executed. On a POSIX host (macOS, and CI's
ubuntu/macOS runners, which do have sh/find/grep/head) both failed:
- The separator-named-file fixture emitted one octal escape per code point, so
U+2028 became `\20050` -- printf reads that as a bare 0x80 byte plus the
literal digits `50`, not U+2028. Dead `len(separator) == 1` branch removed;
the code point is now emitted as one octal escape per UTF-8 byte
(`\342\200\250`).
- The matched-text fixture used `printf '%s'`, which does not interpret
backslash escapes, and interpolated a payload whose own double quotes closed
the shell word; the file ended up as `const s = au2028b;\n` with no real
U+2028. The escape now sits in printf's format string, with the octals
assembled from the code point's UTF-8 bytes.
Verified by running the three cases directly on a POSIX-style root, which is
what the CI runners do: all three pass. The canned-output tests, which pin the
production change, are unchanged.
Also replaced the raw control characters in both changelog entries with
escape-style text (`notes\x0bdraft.txt`, `"a\u2028b"`, `"\n"`): a raw vertical
tab or U+2028 renders as nothing or a replacement glyph, and one code span had
a literal newline in it, which CommonMark collapses to a space and would have
read as `split on " " only`. Matches the #6520 entry's practice of naming the
characters.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
yetuge
|
42df8f3e5f
|
fix(integrations): decode lark-cli and npm output as UTF-8 (#6590)
* fix(integrations): decode lark-cli and npm output as UTF-8 Every text-mode subprocess.run in lark_cli.py that captures lark-cli or npm output omits encoding, so the bytes are decoded with the host locale. Both write UTF-8 to a pipe: the @larksuite/cli npm package's bin is a Node wrapper around a native binary that emits UTF-8 (lark-cli --help writes U+2014 as E2 80 94), and npm output is UTF-8 too. On a host whose ANSI code page is not UTF-8 (cp936, cp1252) the locale decode silently mangles every non-ASCII field -- auth status --json returns a garbled userName, so the status card shows mojibake -- and an undecodable byte raises inside subprocess's reader thread, which is not re-raised: stdout comes back as None or "", so a healthy CLI is reported as unavailable and a successful run raises "lark-cli did not return JSON output." Decode these captures as UTF-8, matching #5440 (PowerShell), #5904 (AIO sandbox docker CLI) and #6548 (Git Bash output in LocalSandbox). The whoami /user capture stays on the locale: whoami is a native tool that writes the host code page, the same UTF-8/native split #6548 drew. Tests drive the real functions with a stand-in CLI emitting UTF-8 CJK JSON and pin the pre-fix decode path, so they fail on main and pass here on any host locale. Fixes #6589 * test(lark): isolate the paths singleton in the encoding suite The new tests reached ensure_lark_cli_credential_tree through the global get_paths() singleton, so every run created and hardened a real <cwd>/.deer-flow/users/alice/integrations/lark-cli/{config,data} tree in whatever directory pytest ran from -- the same runtime home a live deployment keeps real per-user Lark credentials in. An autouse fixture now points the singleton at tmp_path, matching what test_lark_cli_integration.py does. Red/green is unchanged: reverting only lark_cli.py still fails the two assertions that pin the decode, and the suite leaves no .deer-flow behind. * test(lark): make the version probe pin the decode, drop the inert locale patch Two test-rigor notes from review: - test_probe_lark_cli_at_path only asserted available/error, and the probe returns available=True with any non-empty output and a zero exit code, so the test stayed green even with the encoding pin removed from _probe_lark_cli_at_path. The stand-in now answers --version with a non-ASCII payload and the test names the expected text, which fails on the unpinned call site (verified) and passes with the pin. - The locale.getencoding patch was inert: the subprocess.run wrapper injects the codec for unpinned text-mode calls and lark_cli never consults the locale itself, so nothing reached the real locale-decode path. Dropped it, along with the now-unused import, rather than implying a redirection that does not happen. The Windows shim forwards its arguments (%*) so the stand-in sees --version. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
db9b0b4bf0
|
fix(release): reject a stale backend/uv.lock in the version gate (#6588)
* fix(release): reject a stale backend/uv.lock in the version gate verify_versions.sh compared only Chart.yaml, pyproject.toml and package.json, so bumping those three by hand passed the v* gate while backend/uv.lock still recorded the previous root package version. The backend image installs with uv sync --locked, so the chart and the other images published while the backend image failed to build. Run uv lock --check in backend/ as part of the gate, letting uv own the PEP 440 normalization, and fail when uv is missing. The reusable verify-versions workflow installs the uv version the backend image pins. * docs(changelog): reference #6588 * fix(release): explain an unparsed lock version and stop blaming the lock for every uv failure Print a fallback when the lock's root entry is not found, so the job log does not show an empty field, and word the uv lock --check annotation so a network error or uv crash is not reported as a stale lock. |
||
|
Nan Gao
|
6c4c74a504
|
fix(channels): share chat-to-thread bindings through the database (#6558)
* fix(channels): share chat-to-thread bindings through the database ChannelManager kept its IM chat-to-thread bindings (channel_name:chat_id[:topic_id] -> thread_id) in a per-process JSON file, channels/store.json, loaded once and rewritten whole on every change. With several Gateway processes a binding created on one replica was invisible to the others (the next message for the same chat opened a second thread) and concurrent writers clobbered each other's file. - New channel_thread_bindings table (migration 0036_channel_thread_bindings) and harness package deerflow.persistence.channel_thread_bindings: key PK (the legacy composite string), channel_name, chat_id, nullable topic_id, thread_id, user_id, epoch created_at/updated_at; index (channel_name, chat_id). Dialect upsert keeps created_at; prefix remove and batched ON CONFLICT DO NOTHING import. - app.channels.store.ChannelStore is an async protocol: SqlChannelStore for sqlite/postgres, JsonChannelStore (file I/O via asyncio.to_thread) for database.backend: memory. resolve_channel_store picks at ChannelService construction; ChannelService.start() imports an existing store.json once into an empty table and renames it store.json.migrated (a concurrently importing peer neither duplicates nor clobbers; a populated table leaves the file alone). - Callers updated: manager, connection_identity.lookup_thread_id, Feishu (send() awaits the mapping writer; the synchronous lark callback bridges its two store calls to the Gateway loop with a bounded wait), Discord (channels_dir). - Head pins in the 0027_0028/0029/0032/0034/0035 migration tests point at the new head; docs (channels and migrations AGENTS.md, CONFIGURATION.md, README) and CHANGELOG updated. * docs(changelog): link the channel thread bindings entry to #6558 * fix(channels): reject only the Gateway loop in the Feishu store bridge lark-oapi 1.5.5 invokes the message callback synchronously from its async frame handler, on the private event loop that FeishuChannel._run_ws starts for the SDK thread. The bridge's guard treated any running loop as unsafe, so every real inbound message skipped the stored-topic lookup and the pending-mapping restore: a reply whose parent card was mapped but whose root was not fell back to the unknown root, and could consume an unrelated pending clarification. Only a wait issued from the Gateway loop itself (the loop that executes the submitted coroutine) is refused now; the SDK's own loop may block for the bounded wait, as that thread blocked on the JSON rewrite before. Tests drive _on_message from a worker thread running its own loop, the production shape, for both the lookup and the write path, and pin the Gateway-loop refusal. * fix(channels): defer unresolved Feishu lookups and harden the store.json import Review follow-ups on the chat-to-thread binding store (#6558): - Import: a legacy entry whose key, channel_name, chat_id, topic_id, thread_id or user_id exceeds its column width is skipped and counted with the malformed ones instead of failing the whole batch on PostgreSQL (SQLite silently accepted the same row, so the backends diverged on one file). The corrupt-file warning now says what each caller does: the JSON store starts fresh, the import is skipped and the file kept. - Feishu bridge: store calls run on the Gateway loop under a 3s asyncio.wait_for (the SDK loop also services ws keepalives; the old 10s bound could stall it per message), and the lark-thread wait is that budget plus 1s. A deadline or store error is StoreLookupUnresolved, never a miss: _on_message no longer routes such a reply onto its unknown root (which created a second thread and later overwrote the parent alias). The message joins a per-chat FIFO drained on the Gateway loop with backoff for up to 30s, then is dropped with a warning. _on_message is split into parse / resolve / dispatch so the deferred path dispatches the same way; later messages for the chat queue behind the deferred one. * fix(tests): expect follow_redirects in the Jina client kwargs after #5808 #5808 made JinaClient pass follow_redirects=True to httpx.AsyncClient and updated test_jina_client.py, but test_jina_response_limit.py and test_jina_retry_after.py still pin the exact client kwargs to proxy and trust_env, so every backend-unit-tests shard that lands on them fails on main (the backend unit-test workflow only runs on pull requests, which is why the push to main did not surface it). Update the two expectations to include the new keyword. (cherry picked from commit d5aa23156ce19ce36d95df03953442f83ae6d050) * Revert "fix(tests): expect follow_redirects in the Jina client kwargs after #5808" This reverts commit 5bee00bfb09c65a30f77fb4df47a1f82c47ecef7. * fix(channels): bound deferred Feishu routing by the inbound queue capacity Review follow-up on #6558: a store outage let every subsequent message for a chat join the deferred queue with no capacity check (the bus reservation was only taken after routing succeeded), and each queued message started a fresh 30s retry budget when it reached the head, so a burst could be retained for N x 30s and grow without bound. - Intake is reserved BEFORE routing for every store-backed message (bus.reserve_inbound is thread-safe and already used from SDK threads), so deferred and live messages share the one configured bound, channels.inbound_queue_maxsize; a message that cannot be admitted is dropped at once with a warning, never queued or routed as a miss. - The reservation travels with the message into dispatch (committed by _prepare_inbound), is released on the batch path (the flush reserves at publish time), on drop after the budget, on a failed deferral, on dispatch failure and for every leftover when the drain task is cancelled by stop(). - The retry budget runs from the message's arrival: a queued message whose budget is spent when it reaches the head is dropped without a lookup. - The bounded resolver returns the unresolved outcome instead of raising it through the tracked task, which logged an ERROR for an expected, handled store timeout. - Upstream's new blocking-io test constructs the store by its old class name; it now uses JsonChannelStore. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |